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SUMMARY 


This report examines and details the activities of a major U.S. airline during the period 
of late 1993 to late 1997, as it acquired two fleets of advanced technology aircraft, the 
Boeing 757 and the 737-500. The host airline had planned to purchase 767s during 
the period of the study, but delivery was delayed for economic reasons. The 767 and 
757 is considered a single fleet due to the commonality of their cockpits. 

All three aircraft were equipped with electronic flight instrument systems (EFIS), 
colloquially known as "glass cockpits." There are aircraft with flight management 
systems (FMSs), but with traditional instrumentation (e.g. some models of the B-737- 
300). But generally the glass aircraft have both FMSs and instrument panels that are 
driven by computer-based color graphics. These are not simply electronic replications 
of traditional aircraft instruments, but are highly versatile displays that can do what 
traditional instruments cannot (e g. the HSI moving map display, the display of radar 
returns on the map, the display of the wind vector, and the position predictor vector.) 
Prior to the delivery of the first 737-500 in January 1994, the airline had no glass 
airplanes. The most modern aircraft was the 737-300 non-EFIS ("round dial"), with a 
modern FMS (see above). 

Although the primary focus of the study was upon flight training, we examined as well 
the technical support and management of the pilots in these fleets, in some cases very 
detailed matters, such as checklist and procedure design. 

Questionnaire data were collected in three phases: 

Phase 1 - the first day of transition training 

Phase 2 - approximately 3-4 months after transition training 

Phase 3 - approximately 12-14 months after initial operating experience (IOE) 

A total of 150 pilots who were entering 757 transition training volunteered for the study. 
Three were dropped during data analysis for the first stage due to incomplete data 
records. Of the remaining 147, 102 returned data forms in Phase II of the study, and of 
these 99 pilot volunteers also completed the forms in the third phase. 

Face-to-face and telephone interviews were conducted with a sample of 20 line pilots, 
as well with flight instructors, check airmen, management pilots, and ground school 
instructors 

As a side activity, at the request of the company a sample of volunteers going through 
transition to the 737-300/500 was selected and given the questionnaires, before and 
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after a change in the training program. The company wished to have an independent 
assessment of the effect of the change. This study will be reported in a subsequent 
publication authored by Rebecca Chute. 

The 757 study found that by and large, pilots transitioning to the B-757, most of whom 
were going to their first glass cockpit, had high morale, low levels of apprehension 
about the transition, and a generally positive attitude toward their training, and toward 
cockpit automation. They also shared some concerns, such as what they perceived as 
a potential for a loss of basic airmanship skills, and an apprehension about having 
sufficient time for extra-cockpit scanning (“head outside”). These concerns will be 
addressed in this report. 

Start-up Transients 

The program was hampered in the beginning by schedule problems due to uncertain 
aircraft deliveries, at times resulting in insufficient aircraft lines for the number of pilots 
in training. This in turn resulted in pilots having to return to their previous aircraft, or 
other aircraft, before they could later be assigned to the 757. At times the opposite 
occurred - rapid acquisition of aircraft resulted in pilot shortages and an acceleration 
of the training schedule. There was also an unexpected bid off of the 757 due to what 
pilots considered undesirable flying schedules, and their disappointment over the 
cancellation of the 767 order. The 767's ETOPS capabilities and the promise of trans- 
Atlantic flying had been a great motivator for bidding the 757-767 transition. 

The Continental program differed from other programs in many ways, as discussed in 
the body of this report. One significant difference was that preparation for ETOPS 
operations and international flight were built right into the training syllabus. All 757 
pilots emerged with ETOPS LOFT experience. First officers were type rated in the 757- 
767. The 757 was pressed into ETOPS service (with 180 minute certification) very 
soon after the program began. Service began with flights from Newark to Manchester, 
England, and later Newark to Lisbon. In spite of doubts about the marketing issues 
raised by a single-aisle aircraft in trans-Atlantic flight, the 757 was an immediate 
success, both with respect to marketing and flight. Ironically, the 757 flights to Europe 
were so successful that they were taken off the route and replaced by DC-1 Os. 

Pilots entered the transition program with a far more positive attitude toward 
automation, and less apprehension about being able to make it through the program, 
than we saw in previous field studies. We believe that this is due in part to the fact that 
advanced automation, by the time of this study, no longer evoked emotions of 
uncertainty, and, with certain reservations, had proven itself to a skeptical pilot 
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population. Other factors were the generally positive attitudes prevailing in an airline 
struggling to emerge from a stormy recent past, marked by bankruptcies, strikes, 
extremely rapid expansion through mergers and acquisitions, and mistrust between 
management and labor. By the time the first 757 arrived on the property, there was a 
“can do” spirit prevailing throughout the pilot ranks, and the rest of the airline. This 
spirit grew steadily during the years of the study, as we have noted elsewhere in this 
report. Finally, much of the positive attitude can be credited to the respect for and 
popularity of the two fleet managers of the 757/767 program, and the fact that they 
were given a free hand to pick the initial cadre of training pilots. 

The program made ample use of an advanced flight training device (FTD), computer 
based training (CBT), and a full-flight simulator (FFS), modifying the training syllabus 
as they went. Fine-tuning of the training and the use of the devices took place as the 
program progressed. The program had the usual start-up transients. The loss of the 
767 order was a severe blow. 

A Clean Sheet of Paper 

The terms “clean sheet of paper” and "free hand” emerged time and again to describe 
the extent to which the success of the program was the result of unswerving support 
from higher management. These phrases represented not only the all-important 
perception of support, but the practicalities -- that fleet manager’s requests were taken 
seriously, and that management did not quibble or “nickel-dime” the managers of the 
training program. The fact that this support seemed unusual, and needed to be 
commented on, leads the authors to believe that the lot of a fleet manager had not 
always been a happy one. More will be said of this in chapters VIII and IX. 
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I. BACKGROUND TO THE STUDY 


A. CONTINENTAL AIRLINES - SUMMER OF 1 993 


In August 1 993, flight management from Continental Airlines (CAL), based in Houston, 
approached the first author and asked him to consult with them on transition training for 
pilots who would be transferring to the company’s two new fleets of aircraft, the Boeing 
757-767 and the Boeing 737-500. Two weeks later in Los Angeles, accompanied by 
Dr. Everett Palmer of NASA Ames Research Center, he made a presentation and 
proposal to company officials, resulting in this study, a cooperative project between 
NASA, Continental, and the University of Miami. Captain David Lynn, fleet manager for 
the 757-767 program, was named to be Continental’s point of contact for the study. In 
1996 Captain Lynn took over the 737 program, and was replaced by Captain David 
Sanctuary, who remained our point of contact to the end of the study. 

The fleet at the time consisted of the DC-9, MD-80, DC-10, A-300-2B (3-pilot, 
traditional instrumentation, not to be confused with the A-300-600, a 2-pilot EFIS 
aircraft), B-727, B-737-1 00/200/300 (non-EFIS 1 ), and B-747-1 00/200. In addition 
Continental operated Continental Express, flying ATR-42, ATR-72, EMB-120, and 
Beech 1900. The 737 was the largest fleet, and will remain so. The most advanced 
cockpit in the fleet was the B-737-300, with FMC but no EFIS, so CAL had no 
experience with EFIS (“glass cockpit”) at that time. The fleet at Continental proper 
numbered about 460 [note: henceforth we shall consider only Continental Airlines, and 
ignore Continental Express and Continental Connection], The A-300 fleet has since 
been retired, and some B-727s have been retired. Older models of the B-737 are also 
being retired as the 500 models, as well as the “next generation” Boeing 737s, are 
added to the fleet. 

Early in the next century Continental will have an all-glass, all-Boeing fleet. In 1999 
alone Continental will take delivery of 58 new Boeing aircraft. The consolidation 
around Boeing aircraft will result in CAL’s fleet having five, rather than the present nine 
major model types, with predicted savings of $ 50 million per year (Proctor, 1998a). 


1 A glossary of terms, mainly those dealing with flight-deck automation, can be found in 
Appendix B. 
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Under the leadership of CEO Gordon Bethune, Continental will move in a very short 
time from an essentially obsolete fleet to one of the youngest in the industry. 


B. A BRIEF HISTORY OF CONTINENTAL 


Since this study was concentrated at one airline, and one with a turbulent financial and 
labor history through the ‘80s and into this decade, it is necessary to understand some 
of the history of the company. The authors do not take sides in this discussion, but try 
to present an dispassionate discussion and understanding of how the company’s 
background and culture developed, and how it impacted the present study. 

Continental, as we know it today, is the product of many tributaries, including the 
original Continental (“Old Continental” as it is called by pilots), Pioneer, Texas 
International, Frontier, New York Airways, and People Express. Continental was 
founded in 1937 out of a Southern California company, Varney Speed Lines. The 
following year Robert F. Six became president, and led the company for over 40 years. 
He built a California-based company concentrating on providing passenger service to 
the southwest and later Hawaii. In 1953 Continental acquired Pioneer Airlines, with 16 
destinations in the west. Six moved the company from El Paso to Denver, and in 1963 
established the headquarters in Los Angeles. In 1968 Continental formed a subsidiary, 
Air Micronesia, to serve the islands of the Pacific. By 1980 Continental had 180 
aircraft. Today it has earned its position in the “middle three” of U.S. air carriers 
(Northwest, Continental, and USAirways), and is currently engaged in forging an 
alliance with Northwest. 

The Airline Deregulation Act of 1978 ushered in a period of extreme turbulence in the 
airline industry. The experience at Continental was more than turbulent. These were 
the years of “merger mania.” In 1981 , the original Continental was purchased by 
Frank Lorenzo’s holding company, Texas Air. Continental was later merged with Texas 
International Airline (formerly Trans-Texas), but kept the Continental name. It was 
called by many “New Continental.” The following year Robert Six, at the age of 74, 
retired from the airline. 

Lorenzo’s Texas Air Corporation (TAC) which already owned New York Airways, 
bought Continental, then acquired Frontier, New York Airways, and People Express. 

In 1986 Texas Air Corporation purchased Eastern Airlines. In January 1987 TAC 
folded New York Air, Frontier, and People Express into Continental, resulting in two 
companies of about the same size, Continental and Eastern. 

In 1 983 Lorenzo took Continental into bankruptcy. The conventional wisdom was that 
there was no financial justification for his move, that he was using the bankruptcy laws 


1-2 



to defeat the unions. Lorenzo allegedly took advantage of the bankruptcy laws to 
abrogate labor contracts and impose lower wages and longer hours. In the case of 
pilots, this meant more flying time for less pay. The Air Line Pilots Association (ALPA) 
joined the machinists on strike. Later Congress plugged that loophole in U. S. 
bankruptcy law. 

The strike was not well disciplined. Many pilots crossed the picket lines after a brief 
gesture of supporting the strike. The strike ended in 1985, as the company emerged 
from bankruptcy. ALPA was decertified as Continental’s bargaining agent. Later, in a 
certification election, Continental pilots voted to create their own union, the 
Independent Association of Continental Pilots (IACP). This move has interesting 
implications for this study, and they will be discussed later. 

In 1990, Lorenzo was forced to sell his holdings in Continental and relinquish control. 
Immediately following this, management again took the company into bankruptcy. In 
1993 Continental again emerged from bankruptcy, and began the process of 
rehabilitating the airline. 

By 1993, when this study began, there was a spirit of rebuilding, and a cautious 
optimism at Continental. A large fleet of glass aircraft was ordered from Boeing (737- 
500, 757, 767, and later the 777), and they became not only the backbone of a new 
and modernized fleet, but also a symbol of optimism and hope for the airline. The first 
737-500 arrived in January 1994, and the first of an initial order of 757s in May 1994. 
The 767 order was delayed, then canceled, and later reinstated. The impact of the off- 
and-on 767 order on the crews, the transition program, and this study, will be discussed 
later. In 1997 Continental and Boeing signed an agreement for Continental to become 
one of several “all Boeing” airlines. It placed a large order for the 737-700, and later 
became the domestic launch customer for the 737-800. The 767 order was reinstated. 
The first 777 delivery took place in September 1 998. During the year of this writing 
(1998), Continental acquired 64 new Boeing transports. 

Continental, along with most of the larger airlines in the U.S., was enjoying a period of 
prosperity and profits, high load factors, fleet modernization, and an expanding route 
structure (Shifrin, 1998). In 1996 and 1997 Continental was cited time and again by 
business publications and polls as one of the leading examples of a “turn around” 
company, both in its financial success and the quality of its passenger service. Much of 
the credit has been attributed to the leadership style of Gordon Bethune (Bethune and 
Huler, 1998). 
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C. THE HIGH TECHNOLOGY COCKPIT 


The last two decades have witnessed the rapid and wide-spread development of an 
entirely new cockpit technology, based on the capabilities of the microprocessor and 
color graphics. We will not attempt to review the literature, history, or development of 
cockpit automation, as it is well reviewed elsewhere (see Sarter and Woods, 1994, 
Woods and Sarter, 1992; Rudisill, 1994; Flint, 1995; Billings, 1997; Wiener, 1988, 

1989; Wiener and Nagel, 1988; FAA, 1996). For all of automation’s astounding 
capabilities, doubts were expressed about the human factors issues raised by robotic 
flight. For an early version, see Wiener and Curry (1980), and for more recent writings, 
see Last (1997), Learmont (1996), and Foreman (1996). 

Would the average pilot be able to manage the automation and its many modes? 

Would the hardware and software be able to live up to its claims for workload 
reduction, thus making it possible to eliminate the flight engineer’s position, and fly 
large jet aircraft, over oceans, with a two-pilot crew? Would automated flight invite 
operator “blunders” (large errors) as seen in other applications of automation. Would 
pilots “fall out of the loop” and not be able to keep up with the airplane? Would manual 
flying skills become degraded (see Figure 1-1 )? Some of the doubts harbored by pilots 
can be seen in the following report to the NASA Aviation Safety Reporting System 
(ASRS). 

[Note: throughout this report we include, for illustrative purposes, reports selected from 
a search of the NASA ASRS database for cases dealing both with automation and 
training. The ASRS form does not ask for information about the employer of the 
reporter, or identify in any form the carrier(s) involved in the report. If the carrier were 
identified by the reporter, the information would not be stored in the database. 
Accordingly, the reports which are sampled and included in the text and in Appendix H 
are probably not from Continental Airlines, and they may or may not concern B-757 
aircraft. We chose our cases strictly for their subject matter interest, and they should 
not be thought of as reports concerning Continental crews, or even necessarily B-757 
aircraft]. 

NARRATIVE: During IOE training en route PHL to CLE was given clearance to cross 
10 miles east of YNG VORTAC at 24,000'. In discussion with check airman on best 
method to enter this info into FMC, I decided to start down and then work on FMC in 
descent. I inadvertently selected 10,000' into flight guidance system. Again we went 
heads down to concentrate on programming FMC for descent path. Moments later CLE 
center requested our altitude. We looked up as we were through 22,000', leveled out at 
21,000'. We informed Center. Weather was clear and controller just said to maintain 
21,000', apparently there was no conflicting traffic. This is not a new problem. 
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Automation has taken over in the cockpit. Computers are not learned overnight and need 
hands on operating experience. It all comes back to "fly the airplane first!" 

ACCESSION NUMBER: 116912 


Early Studies 

As early as 1977 the first alarm was sounded by the late Elwyn Edwards (1977), who 
examined for the first time the broad question of human factors of cockpit automation. 

At the same time concerns were being expressed in the U.S. Congress. Two 
Congressional reports identified automation as a safety problem for the coming decade 
(U.S. House of Representatives, 1977; U.S. Senate, 1980). There was much talk of 
“the automation problem,” but no person nor any agency was prepared to say with any 
certainty what “the problem” was. In 1979 NASA Ames Research Center was tasked 
with examining the safety implications and human factors in automated flight. The 
Congressional subcommittees had no trouble recognizing the positive side; what they 
wanted to know was whether there was a “down side.” Quite simply, were there also 
adverse consequences of the new flight decks that the manufacturers, regulators, and 
future operators were overlooking? 

The project was assigned to Dr. Renwick Curry, then of NASA Ames, and Professor 
Earl Wiener, on leave at Ames from the University of Miami. Their collaboration 
produced a comprehensive report (Wiener and Curry, 1980) on the human factors of 
cockpit automation, proceeding beyond Edwards’ initial work (1977). They produced a 
list of 1 5 guidelines for the design and utilization of cockpit automation. Guidelines 
from other authors followed (see Billings, 1997). Following the publication of their 1 980 
paper, Wiener and Curry conducted three field studies of the adaptation of the new 
aircraft into the fleets of several airlines (Curry, 1985; Wiener, 1985b, 1989). 

The Advance of the Glass Cockpit 

The decade of the 1980's saw the appearance of the new, electronically sophisticated 
transport aircraft. The Boeing 767 was followed shortly by the 757, and later glass 
derivatives of the 737 and the MD-80. In the 1990’s a family of original aircraft were 
produced by Airbus Industrie: the A-3 19-320-321, the A-310, A-330, A-340, and the 
derivative A-300-600. The A-320 series took automation to a higher level than the first 
generation of glass aircraft, typified by the 757-767 and the A-310, introducing fly-by- 
wire with the side-arm controller and other advanced capabilities. Douglas fielded the 
derivative MD-1 1 , Fokker produced the F-100, and new models of Boeing’s best-selling 
737 soon appeared. The long-haul market today is dominated by a glass derivative of 
the traditional 747, the 747-400, but the smaller A-340 and the B-777 show promise of 
being the dominant long-haul aircraft of the next two decades (Proctor, 1988). 
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The new FMS and glass aircraft were considered a great success. The decision of the 
President’s Task Force on Aircraft Crew Complement (McLucus, Drinkwater, and Leaf, 
1981 ) to allow two-pilot operation of the new jets proved to be wise. This, coupled with 
up to 180 minute ETOPS authority, brought a new era of economical trans-oceanic 


2c. I am concerned that automation will 
cause me to lose my flying skills. 

40 * “ " 



FIGURE 1-1. Concern over potential for skill 
loss, third phase of experiment. 

operations for two-engine, (generally) two-pilot glass cockpit aircraft. Under U. S. 
federal aviation regulations [FARs], for two-engine aircraft, three-pilot crews are 
required for flights over eight hours, and four pilots when the flight is scheduled for over 
12 hours. The success of ETOPS operations was summarized in a news item. 


ETOPS RECORD. Boeing 767 transports have logged more than 1 million ETOPS 
flights with 57 airlines. According to Boeing statisticians, 767 operators now log more 
than 13,000 ETOPS flights a month, many of them across the North Atlantic... Through 
May [1998], Boeing-built twin-engine transports had accumulated more than 1.2 million 
ETOPS flights, according to the manufacturer. Proctor (1998b). 


Doubts and Reservations 

Still there were nagging doubts about human factors. Pilots were evenly divided on the 
workload issue; many interviewed by the authors remarked “I’ve never been so busy in 
my life” [flying the advanced cockpit]. We heard this comment over and over. There 
was genuine concern over not only workload, but potential for skill degradation (“loss of 
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scan” as the pilots call it), though to this day there have been little data put forth to 
support claims of skill loss. In the single study that we are aware of, Veillette (1995) 
demonstrated a significant loss of manual skills in crews flying the automated cockpit. 
Veillette’s work is a worthy beginning, but more investigation of this issue is needed, 
especially as longer and longer flights are anticipated. With 8000-mile legs, and 
augmented crews, one can easily imagine flight schedules in which pilots will make as 
few as one takeoff and landing per month. If their captain is up for a checkride in the 
near future, it could be even less for the other crew members. Figure 1-1 above 
displays the responses to the probe on skill loss. To see the responses during all three 
phases of the study, see page A-2. 

Two comments that we received in our open-ended questionnaire items (Chapter VI) 
were: 


[Flying glass results in] more management and less hands-on. Because of the automation 
being almost fool proof, I tend to hand fly to 15,000 more often. The systems work well, 
but I need to keep basic flying skills in tune. 

I bid off the 757 because of the automation and bad trips. The only thing I really miss 
about the 757 automation is the printer. The old technology is real flying, and it’s fun. 
The old technology makes you a better pilot, by hand flying and using your brain. 


The concern over skill loss and a variety of other factors on the part of one pilot can be 
seen in the following ASRS report. 

NARRATIVE: Descent from FL200 to 12000', using FMC navigation and autopilot. 
Approx 15000' entered tops, encountered moderate to severe turbulence, heavy rain. 
Almost simultaneously ATC cleared to cross 40 southwest LRP, at 12000'. LRP not 
available immediately due not auto select on VOR, off screen on CRT. Captain (PNF) 
scrambled to find the runway chart to get the VOR frequency while I got engine anti-ice 
and ignition turned on. Then Captain began adjusting radar to find out why we were 
getting heavy rain and turbulence. When DME finally locked on LRP, it read 3 1 nm (SW 
of LRP). I deployed spoilers and turned off auto thrust. Rain and turbulence worsened in 
descent. As we approached 12000', I observed airspeed decreasing. Not immediately 
realizing, due to concern about the extreme turbulence, that the autopilot was leveling the 
aircraft at 12000' w/o auto thrust available, I disconnected the autopilot. The aircraft was 
trimmed nose down and continued descent below 12000'. The captain recognized the 
problem immediately and called out, "altitude." Flew the aircraft back to 12000' and re- 
engaged autopilot. Minimum altitude approx 11800'. Contributing factors: proficiency— I 
am junior on a wide body, have been mostly assigned for last 6 months as relief pilot 
(cruise only) or with restricted Captain. Consequently, flew 1 leg in Oct, 2 in Nov., 1 in 
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Dec., none in Jan, 1 in Feb., none in Mar. This was only my sixth leg in 6 months. Crew 
shortage— I am in the middle of widebody transition training (completed FAA oral 3/29. 
Released from training due to a backed up simulator schedule). Due to crew shortage, 
sent out on wide body trip during transition training. ATC procedures-assignment of a 
crossing restriction only 10 nm from the crossing fix, using a navaid which is behind an 
aircraft using FMC equip, imposed an excessive workload on the crew with too little time 
to set it up. Fatigue— I was extremely fatigued after being unable to sleep in the hotel in 
Paris. Hotel is noisy during the day when crew is sleepy, stuffy at night. Company refuses 
to change hotel. (I do not smoke or drink alcohol.) 

Recommendations: The issue of proficiency of relief pilots on long range flights should be 
addressed. Captains in the widebody operation on our airline do not feel obligated to give 
legs to the relief pilot. Once having initiated transition training on new equip, a pilot 
should not be required to operate a previously qualified equipment type w/o at least one 
simulator refresher period. Constant crew shortages are destroying pilot personal lives. I 
am beginning to believe that scheduled airline pilot staffing levels need to be addressed by 
the FAR’s. This is a complex subject, but our pilot group is experiencing intense turmoil 
over the effects of crew shortages. ATC should avoid short range crossing restrictions. 
Controllers should be trained on operational characteristics of FMC aircraft (eg., navaids 
behind the aircraft are not readily accessible). Pilot working agreements do not provide 
adequate leverage to ensure that pilots are given suitable hotel accommodations. 
Unsuitable hotels are second only to crew shortages as the major problem in flight ops on 
our airline. Hotels are changed constantly to reduce costs, and many pilots are 
complaining about fatigue due to inadequate rest on layovers. ACCESSION NUMBER: 
108752 


Likewise the question of relative workload in the automated cockpit is still open. For a 
review, see Wiener, 1993a. A simulator study by Wiener, Chidester, Kanki, Palmer, 
Curry, and Gregorich (1991) compared performance as well as pilot opinion of crews 
operating a DC-9 and its glass derivative, the MD-88, flying the same LOFT scenario. 
The perceived workload was greater for the MD-88 pilots. However, the mean 
differences were small, and this is but one study by which to judge a very complicated 
issue. Is the workload higher or lower in a glass cockpit or a traditional cockpit? This 
question remains to be answered. Responses to two of the attitude probes dealing with 
automation and workload are shown in Figure 1-2 and 1-3 below. Graphs showing data 
for all three phases are on pages A-1 3 and A-8. 

In a bit of irony, the FAA, citing concerns over the workload induced by automation on a 
short leg, stopped Mesa Airlines from flying from Ft. Worth to Houston ( AWST , May 5, 
1997, p. 15). This prompted a letter from DC-9 pilot Bemie Harrigan, who comments 
( AWST , June 30, 1997, p. 6) “I thought such equipment was designed to reduce 
workload. It would take me no more than 1 0 seconds to ‘program’ my DC-9 for such a 
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FIGURE 1-2. Attitude toward workload in terminal areas. 


flight. How far have we come, and where are we going?’ Pilot Harrigan might be even 
more perplexed at an article in which the purchaser of the EMB-145 said that their 
models “will not be equipped with an FMS chiefly because those systems require too 



FIGURE 1-3. Attitude toward automation and workload overall. 
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much head-down time in the cockpit and provides no ‘payback’ during short-haul 
flights” (Phillips, 1997). The industry had been told that automation would relieve 
cockpit workload. The question of excessive head-down time appears in all of the field 
studies and opinion polls that we have mentioned. 

Students of glass cockpit human factors have also been concerned about human error 
rate and severity. Some (see Wiener, 1988, 1989; Woods and Sarter, 1995) have 
hypothesized that automated flight invites rare but large, high-consequence errors 
(“blunders”) by the very nature of digital systems. Results from the LOFT study 
mentioned above did not support this view: the error severity was no different when 
comparing crew errors committed in the DC-9 and the MD-88. 

The matter of mode errors appeared in the training programs to be vexing: pilots 
transitioning to glass for the first time had difficulty understanding and properly utilizing 
the autoflight modes. “Mode error” is a broad term: it encompasses selecting an 
inappropriate mode, not understanding the implications of choice of mode, not realizing 
what mode was engaged, and failing to recognize that a change in mode had been 
made not by pilot selection, but by the FMS. Mode errors were to play a vital role in the 
series of glass cockpit accidents that was to follow (Degani, Shafto, and Kirlik, 1995; 
Degani, Shafto, and Kirlik, in press; Hughes and Dornheim, 1995; Hughes, 1995; 
Phillips, 1995). 

Glass Cockpit Accidents 

In June, 1988 an Air France A-320 crashed while making a low pass over the field at an 
air show in Germany. Misuse of automation modes was blamed. Less than two years 
later another A-320 crashed in Bangalore, India due to mode mismanagement. 
Following this came a string of accidents and dramatic incidents involving first Airbus, 
then other manufacturers’ high-technology aircraft (Sekigawa and Mecham, 1996). 

With the situation appearing to be somewhat out of hand, Aviation Week and Space 
Technology published a two-part series on the automated cockpit, edited by David 
Hughes and Michael Dornheim (1995). We will not comment further on these 
accidents, as they are well covered elsewhere (Hughes and Dornheim, 1995; Billings, 
1997), as well as the official accident reports, most of which have been translated into 
English. For a detailed discussion of human error management, see Reason (1990) 
and Wiener (1993b). We do not cover in this study the expanding area of the effect of 
national and regional culture on accident rates, acceptance of modern technology and 
CRM training. The reader wishing this is directed to Johnston (1993a), and to 
Helmreich and Merritt (1998). 

Starting In December 1995, it was the Boeing 757’s turn to be the center of attention. 
First, an American Airlines B-757 (Flight 965) crashed into a mountain while initiating 
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an approach to Cali, Colombia (Aeronautica Civil of the Republic of Colombia, 1996 [in 
English]). This was the first hull loss accident involving a major, U.S. -operated glass 
airplane. 

Two more 757 accidents followed in short order. The first was a Birgenair aircraft that 
crashed off-shore near Puerto Plata, Dominican Republic (Phillips, 1996b). In October 
of that year an Aeroperu 757 crashed off-shore of Lima (McKinna, 1996b). The Cali 
accident is regarded by many as a turning point in the brief history of the glass cockpit: 
a flight crew without a clear picture of where they were or a clear plan for the approach 
once they accepted a runway change, and over-relying on automation, when hand 
flying and basic instruments would have been sufficient. The U.S. aviation 
establishment noted that heretofore the automation-induced accidents occurred 
exclusively on foreign soil, and were the work of foreign carriers, and mostly foreign 
(Airbus) manufactured. This time it was a U.S. carrier. (At the time of this writing there 
has never been a crash of a large glass cockpit passenger aircraft in the U.S. There 
have been crashes of glass-equipped commuter aircraft.) 

Although there was the inevitable disagreement about the causes of the individual 
accidents, and the role that automation played, and considerable denial on the part of 
the manufacturers (see Hughes and Dornheim, 1995; Dornheim, 1995), more and more 
persons in the industry were willing to admit that there were serious problems at the 
pilot-automation interface, as predicted by Edwards (1977) and Wiener and Curry 
(1980). One concern was the relatively weak role played by the FAA certification 
process in guaranteeing safe designs. In defense of the FAA, it must be recognized 
that certification standards simply did not exist. The certification requirements of the 
Federal Aviation Regulations (FARs), Part 25 were based on an earlier era of 
autoflight, when sophisticated flight management systems (FMSs) were unknown. The 
FAA could not be expected to enforce what did not exist. FAA certification personnel 
were well versed in traditional areas: propulsion, aerodynamics, structures, and 
guidance. They were not prepared for the flight management systems of the 1980's. 

Recognizing the need to develop human factors certification standards for modern 
autoflight, the FAA appointed a committee, chaired by Dr. Kathy Abbott of NASA- 
Langley, and Stephen Slotte and Donald Stimson of the FAA, to study the interface 
problem and make recommendations to the FAA for implementation of certification 
standards. Their report (FAA, 1996) contains a long list of recommendations that will 
form the blueprint for future design and certification of pilot-automation interfaces 
(North, 1998). 

The FAA study in turn brought a flurry of activity in the U.S. and in Europe. The 
research community saw the report as a blueprint for studies that needed to be done to 
support the FAA’s certification effort with timely human factors data. In the U. S., the 
“alphabet” organizations also wanted their influence to be felt. For example, the Air 
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Transport Association (ATA) appointed Captain Frank Tullo (Continental) to head their 
Human Factors Committee. The Automation Subcommittee is chaired by Dr. Tom 
Chidester (American). 

The inevitable question arising out of the accidents is whether glass aircraft are more 
or less safe than traditional models. Boeing produced data (Boeing, 1997; Daily, 1997) 
that showed the mean time between hull loss accidents to be considerably greater for 
glass than for conventionally instrumented aircraft. Confirmation of these results came 
from Airbus (Davis, 1997; Sparaco, 1998). An example of a classic controlled flight into 
terrain (CFIT) accident in an old technology aircraft (Boeing T-43, the military version of 
the B-737-200) which occurred in Bosnia (Phillips, 1996a). 

The data provided by these two manufacturers are difficult to interpret, since the old 
technology planes flew more in earlier years, where many things were different -- less 
safety equipment apart from the flight guidance systems, ATC control and weather 
information were less developed than in the recent 1 8 years of glass cockpit 
operations, and the warning and alerting devices that we know today are fairly recent. 

In addition, the glass aircraft are superior in many ways apart from instrumentation: 
better wings, better engines, and better cockpit procedures, perhaps even the CRM 
movement, to mention only a few. 

The original Boeing data were computed before the 757 accidents. Later the figures 
were recomputed, including the 757 crashes, and the new technology aircraft still had 
superior safety records. 

Perhaps we are asking the wrong question. We should be focusing not on 
comparisons of glass and conventional aircraft, but on recognition of the fact that all 
new transport aircraft will soon be FMS equipped, and will probably be glass equipped. 
The question should be reworded and made more constructive. The question we 
propose is simply this: what can manufacturers, operators, and government do to 
maximize the reliability of human and machine performance of modern aircraft and 
enhance safety? (See McKinna, 1997). 

In this report we shall concentrate on but one aspect of that question, pilot training, and 
in particular training for first-time transition to FMS and glass aircraft. 
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II. TRANSITION TO GLASS 


A. INTRODUCTION 


In this chapter we shall briefly outline the problems encountered when pilots transition 
from traditional cockpits to glass cockpits for the first time, the research issues, and the 
practical decisions facing the airline training community. We shall discuss later in the 
report possible intervention strategies for dealing with these problems. 

Reviewing the literature in automation and training, Wiener noted (1993a) that very 
little has been written on the broad subject of training pilots to fly high technology 
aircraft, and even less on the more limited topic of first-time transition to glass. The 
situation is still, six years later, about the same. A welcome addition is Sherman’s 
dissertation (1997), in which he brings the general automation literature up to date. 

Also of great value are Billings’ NASA report (1996) and book (1997) on cockpit 
automation. Billings remarks (1996, p. 121-122): “Training must be considered during 
the design of all cockpit systems and should reflect that design in practice. Particular 
care should be given to documenting automated systems in such a way that pilots will 
be able to understand clearly how they operate and how they can best be exploited, as 
well as howto operate them.” 

By the late 1990s the aviation community became more concerned about training for 
high technology cockpits, largely as a result of a number of dramatic accidents and 
incidents occurring, first in Airbus, later in B-757 aircraft (Hughes and Dornheim, 1995). 
It is inevitable that following an accident, especially one in which the causes include 
lack of understanding of autoflight modes, that the method and adequacy of pilot 
training in the advanced cockpits will be questioned. The accident occurring in 1995 to 
American flight 965, a 757 on approach to Cali, Colombia was particularly 
incomprehensible, perhaps because of the airline involved. American enjoys a 
reputation of leadership and uncompromising quality in its pilot training. 

Over-Use of Automation? 


The Cali accident exposed to the public some of the hazards of autoflight, and much 
was written both in the human factors literature and the public press about the 
presumed over-use of automation. We will discuss over- and under-use of automation 
elsewhere in this report. But the accident, and the reaction in the press, centered 
around training. Why were the pilots not better trained to use the proper autoflight 
modes, or to revert to manual flight? Why did they not make use of the information 
available (e.g. DME)? We are mindful of Curry’s (1985) plea for “turn-it-off training”, 
made ten years prior to the accident. Criticism of training that over-emphasized 
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automation was coming from all directions. 

Even the usually conservative Avidtion Week end Space Technology spoke up 
editorially. In an editorial in early 1996 ( AWST , February 12, p. 66), immediately 
following the Cali accident, under the title “Failing grade for FMS training , the editors 
waded into the controversy in the first sentence, writing, “The training of airline pilots in 
the use of flight management systems (FMS) is clearly inadequate, and airlines, aircraft 
manufacturers and avionics suppliers should get together to pursue better solutions.” 
Not the usual stuff that Aviation Week and Space Technology editorials are made of. 
Two months later they carried an article by Morrocco (1996) in which he quotes Terry 
P. Newman, a senior test pilot in the U.K. Civil Aviation Authority and member of the 
FAA automation team (Federal Aviation Administration, 1996) as complaining that 
pilots are being encouraged to make use of automation “at every possible opportunity, 
particularly the autopilot, because it can do a better job than you.” 

The effect of excessive confidence in automation has been noted in some accidents 
where the crews are turning to the autopilot in an attempt to resolve a deteriorating 
situation.” In September of the same year they again editorialized under the heading 
“Training is no band-aid for bad design” (AWST, September 2, 1996, p. 228), stating, 
“Unfortunately automation has neither removed human error nor simplified the pilot’s 
job. Instead engineers have used the power of the computer revolution to cram more 
functions into smaller boxes, more information onto displays, and more options into 
flight management systems than the average pilot has any hope of mastering.” A 
comprehensive report by Galante (1995), which included a field study of actual 
performance on the line, identified reasons why pilots “click off” the autopilot, or certain 
flight modes, and continue with a lower level of automation. She did not, however, 
relate these to training. The following NASA ASRS report illustrates the concern with 
over-reliance on automation. 


NARRATIVE: Cruising at FL370 inbound to BDF VOR on J 105 from the southwest. 
Kansas City Center gave us clearance to cross 70 nm south of BDF at FL330. I do not 
clearly recall how far from BDF we were at that time. But we immediately began to 
program our newly fully compliant flight management computer (FMC) for the descent. 
We twice attempted to set up the descent using the full FMC capabilities but were not 
successful, so we then reverted to the more basic FMC capabilities and were in the 
process of starting descent when the controller inquired if we were going to be able to 
make our crossing restriction. He added that we had only 9 miles to go. I immediately 
reverted to a manually controlled descent, i.e. throttles idle, speed brakes deployed and 
maximum rate of descent. We told the controller that we would try to make the crossing 
restriction. I believe that several factors contributed to this incident: 1) this was my 
second trip after being off this aircraft for four months. I had been to wide body recurrent 
training in Oct. and had renewed currency in the simulator. That simulator does not have 
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the "full-up" FMC. 2) my copilot was a qualified Captain on wide body who was on his 
first trip after requalifying, 4 months since his last wide body trip. Both pilots are dual 
qualified, i.e. simultaneous qualification on other equip. 3) we were attempting to utilize 
the new full up features of the FMC, but neither of us were proficient in its use. Nor had 
we been given any hands on training on the new features. 4) we had earlier, with the help 
of a written text, programmed the FMC to cross BDF at FL240 and were not expecting 
the FL330 restriction. 5) we neglected to refuse the clearance when it appeared doubtful 
that we could not comply. 6) we both allowed ourselves to become "mesmerized" by the 
computer programming, which we were both trying to learn by doing. 7) I believe I suffer 
from, as I believe many pilots do, a reluctance to revert to basic skills and methods, 
abandoning the advanced technology in our modem aircraft. That technology seems to 
lure one into a dependence and therefore a state of unwillingness and unpreparedness to 
come to the realization that operating the equipment in the "real world" ATC environment 
is not the same as a sterile simulator. This impression seems to me to be reinforced by the 
"official" insistence that the technology be used as it is an integral and essential part of the 
two man crew concept. I feel that we have neglected to emphasize that the technology 
has its definite limitations in this real world. Had I been more prepared to override the 
automatic features of the flight guidance system I feel we would have had no problem 
complying with the clearance. ACCESSION NUMBER: 59982 


So at least part of the training agenda has been defined as a result of the accidents 
and incidents of the first half of this decade, and the field studies that had uncovered 
the problems even earlier (Curry, 1985; Wiener, 1985b, 1988). It is now clear that 
training for autoflight must include not only proficiency in each autoflight mode, but also 
training on mode selection for the task at hand, and turn-it-off training as well. In order 
to achieve this, not only must training methods and curricula be modified, but 
administrative support for the pilot’s right and duty to use or not use the automation as 
he/she sees fit must be clear. 

Aviation automation practitioners and researchers should note that we are not alone in 
recognizing the potential problem of over-use of automation. The maritime world as 
well suffers from presumed over-reliance on automatic devices. A brief article in 
Professional Mariner magazine (December/January 1998, pp. 68-69) describes the 
grounding of the cruise ship Royal Majesty near Nantucket Island in June 1995. The 
NTSB determined that the probable cause of the grounding was “over-reliance on the 
automated features of the integrated bridge system; Majesty Cruise Line’s failure to 
ensure that its officers were adequately trained in the automated features of the 
integrated bridge system and in the implications of this automation for bridge resource 
management [an adaptation of CRM for ships]; the deficiencies in the design and 
implementation of the integrated bridge system and in the procedures for its operation; 
and the second officer’s failure to take corrective action after several cues indicated the 
vessel was off course.” All of this language should sound very familiar to those who 
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have read aviation accident and incident reports involving high tech aircraft. 

Automation Philosophy 

Relief came in the form of an “automation philosophy” statement (Wiener, 1985a), 
pioneered at Delta, then Continental Airlines, then several others. The Delta statement 
appears in Wiener et al., 1991 . The Continental statement and its development are 
discussed in Chapter VII of this volume, and the various forms of the automation 
statement are in Appendix F. The Delta and Continental statements, and imitators that 
followed, say essentially the same thing: the pilot must be proficient in all autoflight 
modes, but the selection of the mode or modes to be employed (including, presumably 
totally manual flight) rests with the crew. There are, to be sure, practical limitations on 
' this. Federal Aviation Regulations (FARs) require the use of certain automatic features 
for low visibility approaches. Pilot discretion, except in an emergency, stops at the 
doorstep of the FAR. 

Chidester, writing of American Airlines’ approach to the question of authority to select 
modes, said 

“What we are trying to establish in the classroom, and through this article, is to encourage 
our pilots to develop their judgment on how to use the automation on their aircraft. Many 
pilots report feeling pressured to always operate in the highest mode of automation 
available. We need to remove that perceived pressure and encourage pilots to choose 
among the modes in any given situation. To do that, we need to review what has been 
automated, some of the documented effects of automation, and some lessons learned 
(1994, p. 8).” 

All of this translates into a training requirement. It is incumbent on the training syllabus 
to ensure the first requirement, total proficiency in all modes, and to instruct as well on 
the tactics of mode selection. The first task is relatively easy - it is what flight training 
has been for years, only now with modern, extremely flexible, equipment. The second 
task is much more difficult. Not only must the pilot be taught discretionary use of 
autoflight modes, he must be examined and graded on his choices. In a previous field 
study on the 757 (Wiener, 1989), during an interview a captain had this to say 
regarding a simulator check ride: “All my life the FAA examiner has been turning things 
off; now they make us turn everything on.” The problem of autopilot mode errors was 
first pointed out by Wiener and Curry (1980), and has been a popular subject for 
automation researchers (see for example, Degani, Shafto, and Kirlik, 1995; in press) as 
well as the operational community. As we have previously pointed out, autopilot 
modes and their potential for human error has been discussed by many authors, so we 
will not pursue this, except to say that this is a major area of concern for transition 
training, both in the pre-simulator and the simulator phase. 
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B. RESEARCH METHODS IN FLIGHT TRAINING 


We shall note briefly here the various research methods that are available and 
appropriate for examination of pilot training for and transition to high technology 
cockpits. A somewhat more detailed presentation can be found in Wiener (1993a). 

Opinion Surveys 

The opinion or attitude survey is widely used, due to its relative ease of administration 
and analysis. Wiener (1993a) listed ten studies employing attitude measurement. 

Other pilot attitude studies have since been published, including Sherman, 1997; 
Sherman, Helmreich, Smith, Wiener, and Merritt, 1996; Gras, Moricot, and Poirot- 
Delpech, 1994; Rogers, Tenney, and Pew, 1995; Tenney, Rogers, and Pew, 1998; 
Sarter (1991); Sarter and Woods (1993, 1994, 1995); Woods and Sarter (1992); 
Madigan and Tsang, 1990; and Australian Bureau of Air Safety Investigation [BASI] 
(1998). 

Most experimenters have been content to display the results of each attitude probe by 
tabular or graphic means, and perhaps test certain hypotheses using attitude data (as 
in the present volume). These investigators are contrasted with those who have done 
extensive multivariate analyses on their data. Two examples of the latter approach are 
the work of McClumpha, James, Green, and Belyavin (1991), and Sherman (1997). 
Attitude surveys have been criticized for being superficial, and not obtaining “real” data, 
hard performance measures that one would prefer in human factors work. A 
considerable literature has developed defending attitudes as measurement of 
performance; as reviewed by Sherman (1997). In the typical survey experiment, the 
sample sizes tend to run in the area of 100 to 200. Often the population being sampled 
is small by definition (e g. MD-1 1 pilots going through transition training at a certain 
airline). McClumpha et al. (1991) defined a larger population, European pilots from a 
variety of aircraft, leading to a sample of 572, which is at the high end of sample sizes 
so far. Sherman (1997) made use of the vast database constructed by Helmreich and 
his colleagues at the University of Texas, with sub-populations in the thousands (see 
Helmreich and Merritt, 1998; and Sherman, Helmreich, and Merritt, submitted for 
publication). 

The value of pilot opinion data, when hard data such as performance measures during 
a simulator run might be preferred, will never be fully resolved. Opinion survey are 
above all easy to do and relatively inexpensive. In contrast, one simulator session can 
cost as much as $2,400. The data from attitude surveys are valuable per se, for 
example in evaluating a hardware design or a training method, or a general belief such 
as the probe used in this study: “I have no trouble staying ‘ahead of the plane”’. As 
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many authors have asserted, attitude data can be taken as imperfect measures of 
system performance. 


In-Fliaht Observations 

Observations taken from the jumpseat during line operations are the ultimate in realism. 

Examples are Helmreich and Foushee, (1993); Helmreich and Merritt, (1998); Degani 

and Kirlik (1 995); and an extensive study of mixed-line flying of various models of the 

B-737 by Lyall (1990). 

In-flight observations are difficult to come by for a variety of reasons: 

1 In-flight observation requires a trained observer, familiar with the aircraft 

systems, air traffic control, flight regulations, and flight-deck procedures. Human 
factors personnel with those qualifications are rare. Often this problem is 
overcome by using retired pilots as observers, which creates a training 
requirement of its own. Former pilots may be familiar with the environment, but 
are not necessarily good observers, and their expertise in human factors may be 
modest. 

2. Observing, and especially taking notes or logging data, may be frowned upon by 
the crew. It is one thing to have a passive observer in the jumpseat; it is quite 
another to have someone logging data. The airline cockpit is one of the most 
exclusive work environments known, outside of government, military, or law 
enforcement operations, and this is jealously guarded by those who work there. 
Exclusive or not, who among us would enjoy having someone observe our work 
day, occasionally writing something on a clipboard or punching keys on a digital 
device whenever we say or do something, or perhaps when we do nothing? 

3. Cockpit observing is expensive, and possibly inefficient. It is efficient in that the 
experimenter does not have to build or buy anything - the “laboratory” is 
furnished by the airline, ATC, and the FAA. The inefficiency comes from the 
paucity of occurrences of the events that the observer may be looking for, e.g. 
TCAS encounters, certain kinds of errors in using the automation, or perhaps 
CRM behaviors of a specified type. One can fly many legs and never see what it 
is that he is looking for, since it is usually low probability events that are of 
interest. 

4. There is the age-old problem of observation effects. The mere presence of an 
observer may alter the behavior of the crew. This is difficult to overcome, and 
can probably only be accomplished by long-time exposure with the same crew. 
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Observation of Training in a Simulator or Flight Training Device 

Both the simulator and the lower fidelity flight training device offers a highly valid 
platform from which to observe not only the behavior of the crew, but the device and 
the instructor as well. In our research on transition to glass, these observations were 
invaluable. Simulator training was observed in many field studies, including Curry, 
1985; Wiener, 1985b, 1989; Wiener et al., 1991; and Sarter, 1991; Woods and Sarter, 
-1992. 

This approach has some of the same problems as those encounter during in-flight 
observations, as described above. However there is clearly little or no observer effect: 
the instructor absorbs whatever anxiety there is about being observed. The human 
factors observer is insulated from the crew, stationed in the back of the simulator cab, 
out of sight, out of mind. 

The same observations that we made with respect to cost and difficulty of in-flight 
observations apply to the simulator as a research tool: qualification of the observer, 
cost of the device, and rarity of event if the observation is looking for something 
specific. Usually in research into transition to a higher level of automation, the 
observer’s scope is wide-angle. He/she is interested in almost anything that reveals 
what happens when pilots move from low- to high-technology cockpits. 

Experiments in Simulators 

The simulator offers the ideal compromise between the valid but uncontrolled real world 
of line flying, and the highly controlled, but far from valid, experimental booth. The 
simulator’s validity is extremely close to the “real thing,” but it still has drawbacks that 
the experimenter must consider. 

1 . For all its realism, the simulator scenario is still not line flying: no lives and no 
equipment are at risk. As absorbed as simulator pilots may be, they still know 
that they are in a box, on the ground, and no amount of simulated ATC chatter, 
weather, electronic visual scenes, or motion is going to change that. 

2. For the human factors experiment, the extreme realism of the simulator comes at 
an extreme price. In one simulator study of automation effects, (Wiener, et al., 
1991), the study had to buy simulator time (on two simulators - DC-9 and MD- 
88), instructor time, and in addition pay for a pilot-observer. Fortunately pilot 
volunteers served as subjects without compensation. 

3. Airline simulators are not equipped for human factors research. Additional 
equipment to record parameters, sample data, and record pilot inputs may be 
required. (The addition of closed circuit TV cameras in the simulator cockpit, to 
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facilitate LOFT debriefings, has been a boon to the human factors researcher.) 
More and more experimenters are turning to research simulators (such as those 
at NASA Ames and NASA Langley) for their work. These simulators are either 
built ab initio to provide for data collection, or they are retrofitted airline 
simulators. 

4. Following a simulator study, the experimenter is left with a massive data 

reduction task, long before statistical analyses can be performed. It may take 
several person-years to reduce the data to useable form, particularly if the 
variables under study are qualitative (e.g. quality of CRM behavior) rather than 
flight parameters. 

As examples of simulator-based experimentation, we recommend Foushee, Lauber, 
Baetge, and Acomb, 1986; Wiener et al., 1991; T. Abbott, 1995; Sarter and Woods, 
1995; and Veillette, 1995, 


C. DIFFICULTIES IN TRANSITION TRAINING PROGRAMS 


The emergence of the glass cockpit brought a host of training problems, and failure 
rates in transition training heretofore unknown. The typical failure rate that was 
reported in the early and mid-1980s was in the neighborhood of 15 to 17 per cent. 
Dornheim (1992), describing the development of MD-1 1 training by Douglas, quotes an 
a most unbelievable figure of 40 per cent failures in transition to the MD-1 1 in its early 
days. Typically the failure rate in transition training to various models with conventional 
cockpits has been less that one per cent. Something was clearly amiss. After a 
complete redesign of training programs, and the investment in very expensive flight 
training devices, the rate was brought down to about 2 per cent. 

Before continuing, we should take note that the alarming rates no longer exist, and the 
failure rate for transition to glass is in the one per cent range (Wiener, 1993a, 

Dornheim, 1996b). In the present study, only two pilots from the original sample of 148 
failed the 757 transition course, and both were highly unusual cases where motivation 
and personality, not the pilots’ ability, nor the quality of the training program, was the 
clear explanation. 

We shall next discuss a few of the possible causes of the initially disastrous training 
situation encountered by most carriers in their early experience with transition to glass 
cockpit technology. 
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Misinformation and Misconceptions 


Pilots arriving at transition training often came with a stock of misconceptions. There 
was a collection of bizarre accounts of glass aircraft taking over from helpless pilots. 
One such story that swept the airline community tells of an A-320 that entered a 
holding pattern and could not get out, imprisoned by advanced electronics! It did not 
help the trainers that pilots arrived with such accounts, mythical as they were. 

Defeatist Attitudes 


The outrageous stories, along with rumors (some correct, unfortunately) of high failure 
rates, led to pilots arriving for training with attitudes of self defeat. They had also heard 
that the program was impossible -- the popular saying that ground school for the 
transition to glass was like the proverbial “drinking from a firehose." At other airlines 
where we had conducted field studies in the decade of the 1980s, it was said (in one 
form or another) that every pilot on his/her way to transition training had a neighbor 
who had a cousin who worked for a man who lived next door to a pilot who had washed 
out of glass transition training. It was also commonly stated that the older captains 
could not pass the course, due to their lack of computer familiarization, and perhaps 
due as well to the general prejudice about old dogs and new tricks. It is little wonder 
that some captains showed up at the training centers with an over-powering sense of 
impending defeat. Many withdrew their bid and returned to their traditional aircraft. 

Dornheim (1992, p. 93) wrote of his own frustration with his introduction to the complex 
automation of the MD-1 1 . 

The simulator session gave me a rude awakening about the realities of modem glass- 
cockpit aircraft. I expected some takeoffs, landings, approaches to stalls, engine failures 
and other maneuvers. Instead, I received a frustrating walkthrough of the automatic flight 
control system and endless complexities of the flight management system (FMS). I was 
irritated at first, but then I realized that this was what it was all about — pushing buttons 
and memorizing FMS screen pages. 

Dornheim continues in this article to trace the design and even the costs of the various 
training devices employed by Douglas for its MD-1 1 training. Costs of training are also 
discussed in Chapters VIII and IX. 

Poor Curriculum Planning and Implementation 

Much of the blame lay not on the rumor mill, but where it belonged, on those who 
designed the training syllabus. Most researchers who have examined this area agree 
that the basic problem was that the early curriculum planners were hidebound, 
attempting to design their programs as if they were training pilots for the 727. 
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The first author attended 757 ground school at two major airlines in 1986. At one in the 
first session of the first day of ground school, the instructor taught the class CDU 
operations, including how to build “man-made” waypoints. There was virtually no 
introduction to the airplane, at a time where one might have had the opportunity to 
dispel some of the misconceptions, and ease the minds of the students. They jumped 
immediately into the most difficult and unfamiliar parts of transition to glass. 

Some airlines were also poorly equipped with respect to training hardware. At one 
airline, ground school instruction was slide projector based, but they did not have the 
customary cubical and projection screen. Projectors and pilots were lined up side-by- 
side, four to a table, with the projectors pointing toward the wall. At any given time 
there were four projected images, seldom the same, on the wall in front of the pilots, 
which they viewed in coordination with recorded instruction. 

CRM Taught Separately 

The 1980's and early 1990's initiated the era of CRM training (Wiener, Kanki, and 
Helmreich, 1993; Foushee and Helmreich, 1988); the later 1990s witnessed the 
integration of CRM with conventional flight training (systems, maneuvers, navigation, 
etc.) This concept, pioneered by Boeing’s flight training group, was the result of earlier 
misdirected effort, leaving the student with the notion that “real” flight training and 
automation training were one area, CRM was another. 

Boeing’s contribution was to show that the two were inseparable parts of flight training, 
and that both went better when taught and practiced as an integrated whole. This 
integration of the two formed the basis for CAL’s training program. [More is said of this 
in Chapter VII. See also the quotation at the top of page VII-1]. From the first day in 
the FTD to the final simulator session, procedures, actions, and decisions in the cockpit 
were accompanied by communication training (CRM, briefings, etc.). 

Achievements 


One by one, the early problems of transition to glass have been solved. The 
misinformation has abated. Failure rates are virtually zero in the transition programs. 
Captains and older first officers no longer have distinguishable difficulties attributed to 
their age or computer skills. Training program curricula have been vastly improved - 
no more warmed over 727 lesson plans. CRM has been integrated into flight training, 
and this is reflected in the carriers’ AQP applications. Those companies that offer 
introduction to automation courses early in ground school considered them a great 
success. 

The data in Figure 11-1 supports the impression that much of the difficulty had been 
overcome. The data come from the first questionnaire, given the first day of transition 
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training for the 757 pilots (n= 148). About 90 per cent of the pilots either reject the 
probe or take a neutral position. About ten per cent accept the probe, expressing their 



FIGURE 11-1 . Self-report of apprehension upon entering 
B-757 transition program. 


apprehension about the transition program. We feel certain in earlier days of glass 
cockpit transition programs far more apprehension would have been reported. And to 
our collective relief, there are as yet no documented cases of A-320s (or anyone else) 
getting stuck in a holding pattern, beyond the control of the flight crew, imprisoned by 
their automation, and destined to fly all turns right, two-minute legs, until their fuel is 
exhausted and the Law of Gravity takes over. Such a story today would bring laughter 
where it once brought apprehension. 


D. TRAINING CONSIDERATIONS 


In this section we shall discuss some of the factors that must be considered in 
designing and implementing a transition to glass program, whose ultimate worth will be 
measured on the line, not in a simulator. We will consider not only the formal design 
of the program, but various human factors problems encountered in automatic flight, 
such as the potential for skill loss. The list is by no means exhaustive. The factors that 
now must be taken into account, particularly when operating under an approved AQP 
plan, seem endless. This is one of the virtues of the AQP process: it forces the training 
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department of an airline to state its goals, and to perform a detailed analysis of the 
subject matter, as well as the teaching and learning activities required to reach the 
specified goals. 

Understanding Autofliaht 

How much does the line pilot need to know about the overall flight management system 
of the 757? This is not an easy question. Should the pilot merely know how to perform 
the functions he wishes to use in flight, or should he have a larger understanding of the 
overall autoflight system? Sarter (1991 ) is critical of the present training methods, 
criticizing them for a “bottom-up” approach which tells them how to get the job done, but 
nothing about the overall plan and philosophy of the flight management system (“top- 
down” approach). She argues that with top-down training, the pilot would be better 
equipped to solve unique problems, diagnose automation “glitches”, and avoid illogical 
or dangerous mode errors, all of this because they would be able to understand the 
consequences of the modes selected, and other actions and selections in using the 
automation. 

There is no simple answer to the problem raised by Sarter. The pilot must be trained to 
obtain the desired output as a function of his/her input to the automation, and this is 
bottom-up training by any standard. Is it necessary for the pilot to “understand” the 
system? Would the accidents that are discussed by Hughes and Dornheim (1995), for 
example, the A-300-600 crash in Nagoya, have occurred had the pilots been trained 
under a more top-down philosophy, and the crews better understood the consequences 
of their choices? 

Hopkins (1992) states, “Pilots are unanimous in their opinion that training for the ‘glass 
cockpit’ should not be based on the same assumptions which form the framework for 
conventional flight-deck training, yet it still is.” He goes on to quote Captain Steve Last, 
a highly experienced pilot and trainer, who said, “We should avoid FMS training with 
insufficient ‘overview’ at the start; trainees have difficulty later in synthesizing the detail 
to see the whole.” J. Butler (1991) argues, 

“The principles of training for advanced technology cockpits are not dissimilar to those of 
older technology. One of the most important aspects remains to select the right people for 
the task and then to provide the necessary hardware... and training devices to enable a 
rapid and efficient acquisition of knowledge and skill... The fundamentals of the aircraft 
operation must be clearly established, understood and supported by all instructors and 
acquired and complied with by trainee pilots. Training courses, whilst necessarily 
concentrating on the acquisition of flying skills must also place great emphasis upon the 
human factors aspects of team work, crew co-ordination, communication, leadership, 
judgement and decision making.” 
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It is tempting to say that line pilots must not only be able to operate but to understand 
the FMS. But our corporate memory of earlier generations of aircraft and flight training 
should disturb us. It was not too long ago that pilots were taught “everything,” including 
details of how systems worked, specifications and limitations, detailed knowledge about 
systems over which the pilot had no control, and were the concern only of maintenance 
workers. With the coming of the jet age, a new training doctrine arrived: teach the pilot 
only what he/she needs to operate the plane, and leave the rest to maintenance. 

Learning details of electrical, hydraulic, and pneumatic systems and how they work 
together is not the same as understanding the inter-related autoflight modes and how 
they work together. It is only be research, and observation of line experience, including 
accidents and incidents, that we might some day be able to answer the top-down 
versus bottom-up question. In the meantime, the training departments of the world 
must strike a compromise in determining just what level of detail a pilot must master. 

An editorial by AWST (February 12, 1996, p. 66) entitled, somewhat provocatively 
“ Failing grade for FMS training” states, “The training of airline pilots in the use of flight 
management systems (FMS) is clearly inadequate..." The editorial goes on to discuss 
the fact that something of a “cottage industry” has grown up in the airline pilot 
community. This industry supplies unofficial manuals covering cockpit automation. 

The efforts are well meaning, but not welcomed by flight management, due to the fact 
that these manuals are not official, are not approved by the FAA or the customer’s 
company. And they may contain errors. 

Why do pilots buy these products? As Orlady (1991) pointed out, pilots are never 
satisfied - they will always say that they need more training. This was confirmed in 
earlier field studies, and in our interviews and open-ended questions in the present 
study, even though the pilots expressed favorable views of the training program. 

Skill Degradation 


From our earliest field studies to the present we have heard repeatedly from the pilots 
in training for glass, or in their first years of flying glass, of their fear of skill loss due to 
their dependence on autoflight (Curry, 1985; Wiener et al, 1991; and Veillette, 1995). 
McClumpha, James, Green, and Belyvin (1991), in a large-scale survey of attitudes 
toward automation of European pilots, also found the same concern, and reported as 
well that older pilots (> 50 years) and those with more training hours showed less 
concern. Skill loss is a realistic issue. No pilot wants to stand by and watch the asset 
he values the most, his flying skill, deteriorate. At some companies, where policies 
required use of autoflight modes whenever possible, the concern multiplied. Pilots 
found some relief at those companies which developed automation guidelines such as 
those derived at Delta and Continental, allowing the pilots, under most circumstances, 
a choice of autopilot modes (including no autopilot - hand flying). 
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Many pilots adopted their own code of hand flying. For example, when they were pilot 
flying (PF), after takeoff they would hand fly to the first level-off altitude, then engage 
the autopilot; or perhaps hand fly to cruise altitude. Some would hand fly, with or 
without the flight director, at least one ILS approach per trip, weather permitting. The 
list is endless. The self-imposed rules were taken seriously, almost as if they were 
regulations. In a previous field study of B-757 pilots, Wiener (1989) reported that 90 
per cent of his questionnaire respondents stated that they tried to hand fly some portion 
of each trip, and if possible, some portion of each leg. 

Veillette (1995) states four reasons that skill maintenance is important: 

1 . Manual flying skills are necessary to handle the critical flight regime of the jet. 

2. Crews who become task saturated in terminal areas often revert to manual 
flying. [This is a well established behavior that we have seen in other studies, 
both in interviews and questionnaires, as well as in jumpseat observations on 
the line. It should be a discomfort to those who claim that automation implies 
workload reduction.] 

3. Some ATC clearances require a high degree of manual skill, if they are not done 
using autopilot modes (e.g. “slam-dunk” approaches). 

4. Manual handling of the aircraft provides information and situational awareness 
to the pilot. It enhances feedback from machine to pilot. 

ETOPS operations have also had an impact on automation usage, and the skills 
maintenance issue. Most companies which operated twin engine (usually wide body) 
jets such as the B-767 and the A-310 on trans-oceanic flights also used the equipment 
for domestic legs. Some segregated the pilots who were flying the same aircraft into 
two sub-fleets, ETOPS and domestic. Those on the domestic flight had far less 
concern about skill loss. The ETOPS pilots expressed fear that they were losing not 
only manual skills, but automation skills as well, since almost all of their time was spent 
at cruise, with few opportunities to exercise automation skills, particularly CDU 
programming. Some ETOPS trips had a domestic leg at the end. Pilots welcomed this 
as an opportunity to practice automation skills. At the companies that allowed pilots to 
bid both types of trips, pilots who were concerned about skill loss would typically bid a 
line that contained two overseas trips, and several days of domestic flying in a month. 
Again, they took this self-imposed discipline seriously. This may no longer be a 
problem. Most carriers that we are aware of allow mixing domestic and trans-oceanic 
trips. Further discussion of the role of company policies, procedures, and 
implementation can be found in Orlady (1991), Chidester (1994), and Degani and 
Wiener (1994). 
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Unfortunately there has been almost no experimental work on the topic of skill loss in 
today’s automated aircraft, the one exception we are aware of being the study by 
Veillette (1995). What we know about the subject comes from interviews and 
questionnaires. Thus it is difficult to design an intervention if we do not know the 
magnitude or locus of the (presumed) problem. It would be difficult, even if a simulator 
were available, to plan and execute such an experiment. 

“Backward Transition” 


Backward transition refers to the transfer of a pilot from a glass environment to a 
traditional cockpit, and whatever problems this may present. Usually the backward 
transition requires only one or two days of formal training and a simulator check ride. 
The presumption is that a pilot who has been flying glass for some time may encounter 
difficulties if he/she returns to traditional instruments. Once again we have little in the 
way of data: only interviews, attitude surveys, and open-ended questions. (See 
Wiener, 1989, pp. 87-92.) We are not aware of any research that directly deals with 
backward transition. The general sense of what the pilots said was that they had 
trouble at first, but very quickly overcame it, and within a trip or two were up to the level 
of proficiency that they enjoyed prior to their 757 transition. 

The biggest difficulty in the backward transition from the 757 appeared to be the loss of 
the HSI map mode display. Pilots in Wiener’s field study (1989) had expressed a great 
attachment to that instrument, and for better or worse, they had learned to depend 
heavily on it in the 757. The problem emerged in the need to integrate the information 
from various displays to determine one’s position. What pilot would not miss this 
display? While he was training to fly glass, it made navigation, planning, and weather 
avoidance so simple and so precise. 

Specifically, the pilots felt at a loss without the map, and found it difficult to stay “ahead 
of the airplane” without this display. Many said that within a terminal area, either on 
takeoff or approach, if ATC turned them off of the published STAR or SID, they had 
trouble taking into account the various navigation displays in the traditional aircraft and 
knowing where they were. It had been so easy in the 757! Also mentioned was the 
ease with which radar and navigational information could be combined and displayed 
on the 757 map. On the traditional displays, they had to extract radar information and 
then mentally combine it with HSI, DME, ADF, and VOR displays. Again, all of this was 
“done for them” by the glass displays. 

In fact, newly transitioned pilots had to be told not to stare at the map, but to bring it 
into their scan like any other instrument. There should be little surprise that line pilots 
would find this to be the feature that they missed the most when they took backwards 
transition to the 727s, the DC-9s, and the 747s (Wiener, 1989). 
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Backward transition is a topic that is interesting to discuss, but probably not very 
important. Pilots generally do not see it as a problem, at least not after a day or so 
back in the traditional cockpit. Whatever problems are brought about by backwards 
transition seem to vanish quickly, and problem or not, the whole issue will disappear in 
the years ahead, when there is no primitive cockpit to go back to. Were it not for the 
supernatural longevity of the B-727, many airlines would be all glass by now, or close 

to it. 

We raise one more issue before leaving backward transition - the possible loss of 
automation skills during the period the pilot was re-assigned to traditional cockpits. At 
some point this pilot would return to glass and would be expected to have lost some of 
his knowledge and skills in operating in an autoflight environment. The question for 
training departments is how much re-training, and what kind, is required to bring the 
pilot back, hopefully close to the level he had reached in his original glass training. 

Ab Injtio Training and Very Low-t ime New Hires 

In the latter part of the decade of the 1980’s airline service and aircraft began to 
expand, and military flight training began to diminish. The military service, the 
traditional source of airline pilots, looked to be insufficient for the years ahead, 
particularly in Europe and Asia. Other sources, such as flying academies, could not fill 
the gap. So major airlines in both European and Asian countries proposed ab initio 
(from the beginning) training, whereby young men and women, entering the program 
with zero flight time, would be trained, usually at airline expense, up to a point where 
they had their basic licenses (see Glines, 1990, and Telfer, 1993; this topic is also 
discussed in Chapter VIII). At this point students would have approximately 200-300 
flight hours, and large amount of jet simulator time. They would then go through type 
training and join the line. The type could be whatever the airline flew, including heavy 
jets and glass cockpits. 

To the traditional pilot or instructor it may be difficult accept that a low-time pilot trainee 
just out of “primary training” could occupy the right seat in an airliner. In the U.S. there 
was concern about pilot shortages, but ab initio training was never a very attractive 
solution. One fleet manager told us that he doubted that CAL would ever have to hire 
ab initio pilots. CAL hired 880 pilots in 1998, and their mean total flying time was over 
3000 hours. Each had turbine time. CAL, like many carriers with their own commuter 
airlines, will draw most of their new hires from their commuter. In the short run, CAL 
will obtain 1 00 per cent of their pilots from their commuter ranks. Later there will be a 
mixture of backgrounds. 

The anticipated pilot shortage never occurred, primarily because there are various 
ways in which a young pilot can qualify for an airline seat in the U.S. A 300-hour pilot 
would have a difficult time finding employment at even the smallest airlines. 
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Inexperienced pilots in the U S. have a hard life: they must take any kind of flying job 
(usually instructing beginning students) in order to build up their hours. The next step 
is usually the air charter or Part 1 35 operator, which serves as a “farm club" for the 
larger regional carriers and the major airlines. How long he/she stays at a farm club 
depends on the market. There is some movement toward the pilot applicant, not the 
airline, to carry the cost of primary training. Some carriers, such as Southwest, require 
new hires to have a type rating, in this case for a 737. The airline hiring the pilot may 
provide, at a price, the necessary training, or the candidate may go elsewhere to obtain 
his/her type rating. 

Finally, let us consider the following question. Given a low-time, zero jet time ab initio 
graduate who is recruited by an airline. If the airline has its choice, where should 
he/she be placed for the first line experience, the traditional cockpit or the automated 
model? The traditional cockpit is simpler, and more like the aircraft the applicant had 
trained in. Remember, his/her exposure to autoflight is almost nil. The 737 or DC-9 
sounds just right. On the other hand, it could be argued that the very low-time pilot is 
best off in a highly automated plane, with a sophisticated autopilot and autothrottle 
supporting him. Only line experience and research will answer the question that we 
have posed. 

Airline training departments may find it difficult to believe that they will ever hire ab initio 
or other very low-time pilots. But who would ever have dreamed that over half of the 
graduating class from the Air Force Academy, due to the cutback in flying, will now go 
to non-flying jobs? 

Advanced Maneuvers Training 

Following a series of airline mishaps and close calls in the latter half of the 1990's, in 
which the aircraft became severely upset and had to be recovered from an abnormal 
position, some airlines instituted “advanced maneuvers” simulator training for all pilots. 
Military pilots for the most part had such training (traditionally called “unusual 
positions”), but those who came to the airlines from civilian sources often did not. This 
training is required at Continental. For a comment from a pilot, see the ASRS report at 
the end of this chapter. [Note: advanced maneuvers are not part of this study, as the 
program came after our work at Continental was complete]. 

Training Devices 

We shall mention training devices only briefly, as so much has been written on this 
subject. The last two decades has witnessed a rapid development of training devices, 
both at the high end, the full flight simulators (FFS) and what will probably soon be the 
middle on the sophistication scale, the flight training device. The FTD provides pilot 
trainers with a device with full systems and flight simulation, including autopilot and 
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flight director modes and glass displays. In the interest of economy of both purchase 
price and maintenance, it does not have a visual scene or a motion base. FTDs offer 
the pilot trainee an excellent platform upon which to obtain cockpit familiarization, 
including running checklists, cockpit procedures (normal and abnormal), flight 
maneuvers, and autoflight modes. For a discussion of the importance of the FTD in 
one program (MD-11), see Dornheim (1992). 

The biggest problem is that FTDs are coming at a steadily increasing price, due largely 
to the number and complexity of autoflight modes that they must simulate. One airline 
that participated in an earlier field study (Wiener, 1989) found that their FTD, a very 
elaborate model, was converging in price on the FFS. They canceled their order for a 
second FTD, preferring to put their funds toward a second FFS. Better to pay more up 
front in order to have the sophistication and regulatory status that only a full flight 
simulator enjoys, they reasoned. 

At Continental the 757 FTD (Level 5 out of 7 on the FAA’s rating scale at this writing) 
and the computer based training (CBT) were carefully integrated into a logical syllabus. 
The typical ground school day is: two hours instructor briefing, two hours FTD, and 
four hours CBT. After two weeks of this, they move to the FFS. (At this writing, 
Continental’s FFS is level C on the FAA scale, soon to be upgraded to level D, the 
highest level). Training emphasizes not only the technical material that had to be 
mastered, but also checklists, procedures, communications, briefings, and CRM. 

What seems to be missing in the array of training devices would be a device so small 
and so inexpensive that it could be provided, along with the software, to each pilot, not 
only for transition training, but for recurrent, and for incorporation of new devices. One 
can recall the confusion that existed over the training for TCAS, and the argument 
about whether TCAS training had to be in a simulator. We asked in a previous field 
study (Wiener, 1989) why the personal computer, perhaps as a home study aid and 
motivator, could not be used to relieve the load on the training center, and particularly 
on the FTD. This subject is also discussed in Chapter VIII. 

Recently there has been some developmental work on using an ordinary laptop 
computer as a flight training device. Stephen Casner of NASA-Ames has programmed 
a relatively inexpensive laptop as a B-737 CBT device, with highly attractive color 
graphic displays. Nordwall (1995) describes how the U.S. Navy is using laptops for 
pilot training. He writes (pp. 68-69), “The capability of the new CDNU (control display 
navigation unit) exceeded Navy expectations. Its use has broadened from a tutorial aid 
to something pilots can use for proficiency training and dynamic simulation.” 

Clearly the potential exists for development of very sophisticated, low-cost personal 
computer-based devices. The problem, as always, will be the cost of development and 
distribution of software. One could envision software upgrades being included in the 
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pilots’ Jeppesen revision envelopes. Certification of PC-based training software is 
something that the FAA presently is not well equipped to do. Presently there is also 
the beginning of a discussion of “web-based training", whereby software can be down- 
loaded from the world-wide web (WWW) to a personal computer. This has two 
advantages over conventional PC-based systems: the cost of delivering software is 
reduced; and the uniformity of software is assured. Web-based learning combines the 
advances of PC-based training with a highly efficient, quality controlled means of 
delivering and updating software. 

Cost of Training 

We shall mention only briefly the matter of cost of training. See also Chapter VIII. We 
have previously discussed the rising cost of flight training devices and simulators, but 
simulators, while dramatic, are only a part of the picture. Cost considerations in 
designing a training program cannot be ignored, in the highly competitive and cost 
conscious economic environment of post-deregulation operations. Gone are the days 
when the word training was sacred, and training departments could get anything they 
wanted by waving the flag of safety. Today every cost in the training process must be 
justified, and the justification may be a traditional one drawn from the corporate world, 
return on investment (ROI). Kelly, Graeber, and Fadden (1993) discuss the ROI 
principle in flight-deck design: “While many operational features provided by a flight 
deck may be considered desirable, the market increasingly demands return on 
investment for capabilities as opposed to features (p. 56)” The same statement could 
be made about training capabilities, though the direct connection to the market place is 
somewhat less visible. Cost figures are seldom published. An exception is 
Domheim’s article (1992) on the MD-11 school at Douglas. 

Orlady (1991 , p. 2.6) cautions us about assuming that automation can reduce training 
requirements: “Unfortunately, one of the great myths of automation is that automation 
reduces training needs. One of the persuasive arguments for further use of automation 
has been that it reduces training costs. This assertion is patently false, particularly in 
the areas of manual skills, system knowledge, and the logic of the automatics.” 

On the other hand, Leonard (1993, p. 149) states that when CBT training for the 
advanced cockpit is combined with an FTD, “The results have been an overall cost 
reduction in flight crew transition training and an increase in successful training rates 
for advanced flight deck aircraft.” He quotes failure rates of training for the glass 
cockpit, in 1984, as 40 per cent. He describes the problem in economic terms, saying 
(p. 150), “This failure rate was unacceptable because extensive remedial training of 
flight crews was economically unacceptable. The high failure rate dramatically 
highlights the inadequacy of existing training strategies to develop the cognitive skills 
required by evolving aircraft technologies.” Leonard goes on to describe the 
development of the MD-1 1 training package. 
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Crew Resource Management (CRM) 


We will only mention CRM training briefly here, as Chapter VII is dedicated to CRM at 
Continental. Crew resource management, or alternatively cockpit resource 
management, was first explored by the airlines in the 1970 s, and was developed as a 
commercial product and sold to other airlines by United. Not until the early 1980's was 
it widely used or explored (Cooper, White, and Lauber, 1979; Wiener, Kanki, and 
Helmreich, 1993). CRM training was not an FAR requirement, but its presence as an 
FAA advisory circular (AC 120-51 B) is a clear signal to the air carriers that the FAA has 
more than a casual interest in this form of training. The AC is usually a precursor of an 
FAR. Under the Advanced Qualification Program (AQP), CRM is part and parcel of the 
training and evaluation. Not only must CRM be included in the training proposal, but 
pilots will be evaluated on their CRM skills. Continental was one of the pioneers of 
CRM training; their training approach and materials have been widely imitated. 

As we have mentioned elsewhere in this report, the trend today is toward integrating 
CRM training and technical flying training. This approach has been developed and 
encouraged by the mainframe manufacturers, and was employed by the 757 planners 
at Continental in their design of the training syllabus. Under the new approach, no 
longer will pilots be exposed to stand-alone CRM training in the classroom. CRM will 
be taught in the FTDs, the simulator, and in conversation with the instructors, as a 
subject intermingled with traditional maneuver and procedure training. In each 
maneuver, or checklist, the CRM aspects will be taught along with the technical 
training. AQP programs also encourage this type of training. More will be said of this 
in Chapter VII. The importance of learning and practicing good CRM skills can be seen 
in the ASRS report below. 


NARRATIVE: The problem began approx 100 nm south of DC A. The captain was 
flying. I obtained the ATIS, LDA-DME 18 was in use. That approach is not in our 
FMS's database so I started to build it. The captain told me not to do that. His 
explanation was that is one of our simulator scenarios, to check on CRM ability, and he 
wanted to practice it first. About 5 mins later I stated my concerns about not using all the 
equipment at our disposal. The navigation display is a great help and we were not going 
to use it. The captain restated his wishes and I dropped it. While being vectored for the 
approach, I idented the localizer, however we were not receiving the DME. As we were 
being turned onto final the captain instructed me to reconfigure my panel to get the DME. 
This leaves me with no localizer indication and no navigation display. By the time I did 
this, we received another turn and a descent, 3000 ft down to 2000 ft. Then I noticed the 
autopilot was not set to capture the localizer. I pointed this out. The captain armed the 
autopilot. He is new (3 months) to the airplane and was behind. As we descended 
Approach asked if we had final OK. The captain lied and said 'yes ’ just then we broke 
out. I saw the river to the right and pointed it out. Approach once again asked if we had 
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the final approach course and gave us a turn to the right. That was very quickly changed 
to a left turn and climb to 3000 ft. We accomplished this, did the after takeoff checklist 
and followed more vectors. My situational awareness was shot. I offered once again to 
build the approach but the captain refused. We were once again vectored to final and I 
asked Approach what was our relationship to the localizer. We were already through it 
and getting worse. Approach Control broke us off once again for the VOR 1 8. We asked 
for time to review it and set it up. This approach was successful. There was no CRM. 
The equipment on board was not properly utilized and I was not properly utilized. No 
matter how much CRM training is given, some people don't get it at the most basic level. 
ACCESSION NUMBER 110413. 


E. CONCLUSIONS 


What can we conclude at this point in the report? It is clear that many of the problems 
that were experienced in the early years of the glass cockpit have been overcome. As 
we noted, the failure rate today in first-time glass transition is about the same as in 
traditional aircraft, less that one per cent. But pass/fail does not tell us the entire 
picture. We must certain that those 99+ per cent who graduate and transition to glass 
do so with the training that will serve them well in their line flying. The line is the 
ultimate test. 

The most striking criticism charges that flight training is not governed by any overall 
philosophy. Perhaps the final product is an amalgam of philosophies, some 
complementary, some antagonistic. It is essential today, and will be more essential in 
the future, that the training package for any aircraft be consistent not only with the best 
operation of the equipment, but with the objectives of the company and the training 
objectives of the entire fleet. With a variety of aircraft flown as a common fleet (e.g. the 
many models of the B-737 that are presently available and will soon be on the line), a 
unified training philosophy is essential. 

As Degani and Wiener (1994) observed, procedures are not strictly determined by the 
hardware: the same piece of equipment is operated according to different procedures 
at different carriers. The procedures are governed only partly by the hardware, but 
also by the philosophy, background, mission, history, operations, and corporate culture 
of the company. Some differences are trivial (e.g. various ways to set up TCAS 
modes), some are dramatic. For example, at the beginning of the study, only one major 
U.S. airline that we are aware of employed QFE altimeter procedures. What was it 
about the flight culture of that company that they, and they alone, found it desirable to 
use QFE altimetry? In 1998 the company abandoned the use of the QFE altimeter 
(personal communication, T. Chidester, 1999). 
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Degani and Wiener interviewed top flight management, starting with the vice-president 
for flight, at three major airlines on procedure development, and asked, “why do you 
develop your own procedures? Why not just follow the Boeing (or any other 
mainframer’s) procedures?” The answer was always the same. Boeing designs and 
assembles aircraft. We fly passengers.” 

We believe the 4P’s model of Degani and Wiener could be used profitably in designing 
training: AQP may have already forced the issue. Their model states that philosophy 
determines policies , policies lead to procedures (or in the matter at hand, training 
packages), and procedures are compared to practices (what actually occurs). Without 
the unifying influence of the first two P’s, training programs are likely to be a hodge- 
podge. A philosophy-based training program could avoid this, and meet the critics 
charges that most training programs are based on anything but a unified philosophy. 
More likely training programs are based on tradition, convenience, cost-containment, 
and the whims of dozen or more training directors, fleet managers, and newly minted 
AQP specialists. 


PHILOSOPHY 

POLICIES 

PROCEDURES 

PRACTICES 

Table 11-1. Degani and Wiener’s “Four P’s.” 


We end this chapter on a happy note, an ASRS report where the pilot claims to have 
saved the day and gives credit to his “advanced maneuvering” training. 


NARRATIVE: Conducting a visual approach to runway 23L sidestep runway 23R at 
MEX. At about 3000 ft AGL aircraft encountered unexpected wake vortex. Aircraft 
rolled rapidly to right approximately 45- 55 degrees. Recovery initiated in accordance 
with company training for advanced maneuvers. 2 minutes later we were told of a heavy 
Airbus landing on runway 23L. Had Mexican ATC warned of the Heavy, I could have 
flown above the glideslope. Callback conversation with reporter revealed the following: 
Report was used for structured callback and following information was obtained. 
Reporter had just completed the new program initiated by his company for advanced 
maneuvers training. The experience with the Airbus was almost identical to the simulator 
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training he had completed. He said it was almost like a time warp where for a nano 
second he felt he was back in the simulator. He feels that is why he handled the situation 
so well and with very little stress. He feels strongly that all air carriers should institute 
such a program ACCESSION NUMBER: 307029 
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III. STUDY METHODOLOGY 


A. BASIC QUESTIONS AND PREMISES 


Our basic premise in designing this study was that information on the detailed features, 
as well as an assessment of the quality of the training program, could be obtained by 
seeking data directly from the pilots involved. For example, information could be 
gained from questionnaires (attitudes, experiences, etc.), interviews, flight-deck 
observations, and direct observation of the ground school training. All of these were 
essentially subjective measures; we would like to have more objective measures, but 
these do not exist, or cannot be obtained at a reasonable cost, in most training 
programs. Even instructors’ evaluations of maneuvers, or overall simulator 
performance, are essentially subjective. The fundamental information upon which this 
report is based comes from pilot responses to questions (interviews) and questionnaire 
data, as well as the authors’ observations. For a brief review of the attitude surveys 
related to cockpit automation, see Wiener, 1993a. The list of studies has grown since 
that writing. 

Longitudinal Studies 

This study is essentially anthropological. We did not manufacture conditions or 
manipulate independent variables, as the experimenters did in a previous automation 
study (Wiener et al., 1991 ). Like anthropologists, we accepted the “village" of transition 
training as we found it, attempted to learn something about the culture, and sought to 
be as unobtrusive as possible. With the exception of our interviews and three 
questionnaires, we generally achieved unobtrusiveness. 

This experiment was designed as a longitudinal study. A longitudinal study is simply 
one in which two or more sets of measurements are taken over the same sample during 
the span of the experiment. This allows the analysis to include not only absolute 
levels, but to evaluate change, or the effect of time, or other interventions. This is to be 
contrasted with a cross-sectional design, in which each sample is observed only once. 

The advantage of a longitudinal study is its sensitivity to change. In this and many 
other human factors studies of cockpit automation, the primary tool is the attitude scale. 
A more objective dependent variable, one that is a sensitive measure of the strength 
and weaknesses of automation, is desirable. Such measures are seldom available. 
Even in costly and time-consuming simulator studies, there are seldom any objective 
dependent variables to measure, and the experimenter again turns to subjective 
measures of automation effects. For an example, see Wiener, et al., 1 991 . In that 
study, an attempt to measure the effect of automation on the communication of crews 



flying a LOFT scenario, the independent variable of automation was achieved by 
collecting data from crews flying the same LOFT in two models of the same aircraft: the 
traditionally configured DC-9-30, and its high-tech (glass cockpit) derivative, the MD- 
88. Even here, with full simulation, and a highly scripted LOFT, we ultimately had to 
rely on observers and simulator instructors to provide the raw data of the analysis. For 
a thorough discussion of this question, we recommend Gregorich and Wilhelm (1993). 

While sensitivity to change over time is the strong point of longitudinal studies, there 
are also weak points and disadvantages. The more serious disadvantages are cost, 
and loss of subjects from the original sample. The experiments are costly because they 
must be continued over time in order to obtain two or more data collection points (which 
we will refer to a “phases” in this study). In this experiment there were three phases. 
The greatest hazard in longitudinal analysis is the steady and unavoidable loss of 
subjects. Subjects lose interest and drop out, simply do not fill out questionnaires, or 
fail to appear for interviews for a variety of reasons. They may retire, die, or be 
reassigned, become medically disqualified, or most likely, change address and fail to 
notify the experimenters. 

In some experiments (such as this), subjects who drop out for reasons of reassignment 
may be of particular value. In the present experiment, these were pilots who at least 
completed the B-757 transition program, and then were reassigned to other aircraft, for 
administrative reasons related to new aircraft delivery. We were eager to speak with 
these pilots because of the shortage of information on “reverse transition”, going from a 
modern aircraft back to a traditional cockpit (see previous chapter, pp. 15-16, for a 
discussion of “backward transitioning"). 

The effect of a pilot receiving training and possibly line exposure to the glass cockpit 
and then returning to the traditional models is worthy of study: it happens every day, 
and we know little about it (Wiener, 1989). Questionnaires 2 and 3 (Q2 and Q3) 
contained a question specifically for pilots who had made a reverse transition (see 
Appendix D). Some pilots in the early days of the 757 program were sent back to their 
former planes to await available seats in the 757. As aircraft deliveries accelerated, 
pilots quickly moved back to the 757 line. 

The human subject in a long-term experiment is not an inanimate object or a lab 
animal. There are inevitable changes in his/her existence that, quite apart from aircraft 
training and line experience, affect a pilot’s lifestyle, flying habits, and certainly on-the- 
job attitudes. The volunteer in a longitudinal study, particularly a pilot in a highly 
dynamic industry, is a moving target. 
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Organization of This Report 


Chapters in this report are designated by Roman numerals. Each chapter has its own 
page numbering, in Arabic numerals. Thus page V-7 is the seventh page of Chapter 5. 
The figures and tables are likewise numbered. Some of the graphics will not be 
assigned figure numbers, since they already have internal numbers as part of the label. 

The appendices are in Chapter XII, and are designated alphabetically, again each with 
its own page number (e.g. D-4). References are based on authors’ names, and year of 
publication (e.g., “Sherman, 1997") and are listed in Chapter X. 

Basic Design of the Study 

This study was designed as a three-phase longitudinal experiment. The phases are 
data collection points, in time, which we designate as “PI", “P2", and "P3". The three 
phases are based on the pilot volunteer’s entry into the program, as follows: 

PI : The first day of transition class (ground school). 

P2: Approximately 3-4 months later, a time at which a pilot will have completed 

transition training, including ground school, simulator, and initial operating 
experience, and will have started line flying. If there were no delays in 
assignment to the 757, the pilot should have a month or two of line flying 
following IOE before receiving his P2 questionnaire form (Q2). 

P3: Approximately 12-14 months after P2, when the pilot would have about 700-900 

hours of line experience in the 757. 

The location in time of PI was fixed: the first day of transition training. A package 
consisting of the first questionnaire (Q1) was distributed by the 757 fleet manager at 
the beginning of the first day of ground school. He encouraged the pilots to sign up. 
The package included the first questionnaire, instructions on how to sign up, and a 
sign-up sheet with informed consent form, and a description of the confidentiality 
protection. The questionnaire can be found in Appendix D. Most pilots who signed up 
did so on the spot, filled out the questionnaire and mailed it to the authors at NASA 
Ames. The confidentiality system is discussed below. No effort was made to contact 
those who did not join the study at PI . 

150 pilots signed up for the study using the blank in Q1. Three who signed up were 
later dropped due to incomplete data. Pilot volunteers were considered members of 
the cohort when their Q1 arrived at NASA Ames. They were mailed Q2 approximately 
four months later, and Q3 approximately 12-14 months after that. Those who did not 
respond to Q2 were still sent the Q3 form. 
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It is possible that a pilot could fill out the forms in the wrong order. That is, he could 
store Q2 and not send it in until after Q3 had been filled out. Although there was 
considerable delay in some of the Q2 questionnaires arriving, we have no reason to 
believe that any were responded to out of order. 



Figure 111-1 . Participation (number of questionnaires completed) 
by phases of the experiment. 


Thus the delivery of Q2 and Q3 was individually timed to each pilot’s entry into the 
program. However, the time at which the form was filled out and mailed back to us at 
Ames was completely under the pilot’s control. 

In summary, the basic experiment consisted of three applications of questionnaires, 
including among other questions, an attitude scale of 20 to 24 items (“probes”). A 
perfect record would be a pilot volunteer filling out all three, and returning them in a 
timely manner for inclusion in the database. 

Training Books: Data Discontinued 

The most objective data that we had available were grades as recorded in what 
Continental calls training books. These are essentially grade books, with pages in the 
same order as the training syllabus. At the end of each graded session (e.g. simulator 
periods) the instructor would enter the grade. The pilot trainee retained his own book 
as he progressed through the program. When the training was completed, the 
necessary sign-offs were made in the book. When everything was complete and 
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satisfactory, the pilot delivered the training book to his chief pilot, and at this point the 
pilot was ready for initial operating experience (IOE) and line qualification in the 
airplane. The training book was destroyed. No grades were kept -- only the fact that 
the training was completed. 

The training books were made available to the project. Unfortunately they proved of 
little value, since nearly all of the grades recorded were the same. This was due to the 
fact that in airline training, the company is looking for standardization and conformity to 
a standard. It is not looking for unusual virtuosity. This is somewhat embedded in the 
nature of airline pilot training. A maneuver can be done satisfactorily or 
unsatisfactorily; it cannot be done “beautifully.” If some grades were unsatisfactory, the 
maneuver was repeated. Given the skill and motivation of the trainees, and the 
instructional skills of the trainers, it is not surprising that nearly every grade was 
“satisfactory.” Therefore, to our disappointment, the training books were of no value for 
statistical evaluation, and were destroyed. 


CAL’s grading system 

1 Close to perfect 

2 Excellent 

3 Average 

4 Satisfactory, but needed to be repeated 

5 Unsatisfactory 


B. QUESTIONNAIRES: ATTITUDE SCALES, 
DEMOGRAPHY, AND FLYING EXPERIENCE 


Questionnaires 


The questionnaires are included in Appendix D. Q1 and Q2 are reproduced in toto ; Q3 
is essentially the same as Q2, except for some demographic questions, which can be 
found on page D-1 0. To conserve space, the parts of Q3 common to Q2 were not 
replicated in this report. 

The questionnaires consisted of three parts: 

1 . A Likert-type attitude scale (see example below) dealing with opinions about 
flight safety, piloting, and particularly cockpit automation. There are 20 items 
(“probes”) in Q1 and 24 in Q2 and Q3. Certain items in Q2 and Q3 were 
inappropriate for Q1 , since the pilots had not yet taken their 757 training. 
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2. Demographic data, mainly questions about past flying experience, but also 
questions about age, computer usage, and aircraft preferences. 

3. Open-ended questions which allowed pilot volunteers to express their beliefs 
and feelings in their own words. These were read and classified somewhat 
subjectively by the experimenters (see Chapter VI). No statistical treatment was 
performed on these data. 

4. Q1 was both a recruiting and data gathering instrument. It contained a 
description of the study and a sign-up sheet, and an informed consent form. In 
addition it contained a 20-item Likert scale, in order to measure pilots’ attitudes 
as they entered 757 training. Demographic information was also obtained (see 
Chapter IV). 

With the loss of the anticipated data from the training books, we had to rely more 
heavily on the questionnaires, interviews, and direct observations. The raw data from 
the questionnaires were entered into a computer database at NASA Ames, and data 
files were sent to the University of Miami for analysis. Statistical analysis and graphics 
design were performed using the SPSS for Windows 6. 1(a) ™ package. Most of the 
demographic data are displayed graphically in Chapter IV. 

Likert Scales 

Likert scales for measurement of attitude are in wide use. They are easy to design, 
easy to administer, and the format is generally familiar to the population being sampled. 
In brief, an item consists of a statement (“probe”) which can be positively or negatively 
stated. The respondent replies by accepting one choice of a multiple choice of items 
showing the agreement/disagreement with the probe, and the degree of this sentiment. 
This is called an intensity scale: the respondent states not only whether he/she agrees 
or disagrees, but the intensity of this belief. Usually there is an odd-number of choices, 
and the center is one of neutrality. The center choice is somewhat ambiguous: it could 
possibly mean “no opinion”, “undecided”, “don’t care”, or a truly neutral position on the 
content of the probe. 

For a summary of the results of several studies employing this technique to measure 
attitudes toward cockpit automation, see Wiener (1993, pp. 216-220). Since the 
publication of the first review there have been more such studies, including a large- 
scale sampling of U.S. air carrier pilots by Sherman (1997). 

Likert data can be treated as coming from a nominal scale (“strongly agree” and 
“agree” are simply categories of response, having no ordinal or numerical relationship 
to each other, or as an ordinal scale , meaning that the responses to the probe could be 
put in a logical order: “strongly agree” is stronger endorsement of the probe than 


111-6 



“agree”, which is stronger than “neutral”, etc. Many experimenters treat Likert 
responses as if they are from an interval scale , attaching numerical values to the 
responses. For example, “strongly agree” would be scored as a “1", “agree 1 as a 2 , 
etc., and the results handling statistically as if interval scores had been generated. The 
problem here is that the numbers and intervals are entirely arbitrary: using 1 ,2, 3,4,5 as 
numerical values treats the distance between responses as psychometrically equal: the 
distance in attitude intensity between “strongly agree" and “agree” would be assumed 
to be the same as between “agree” and “neutral," a questionable, though oft-made, 
assumption. 

The Likert data are displayed as in the example below at various places in this report. 
The entire set of graphics are displayed three to a page (Q1, Q2, and Q3) so that the 
reader can view longitudinal differences. The graphs are found in Appendix A. 

Demographic and Flying Experience Data 

All three questionnaires contained questions of a demographic nature; most dealt with 
flying experience, at Continental and elsewhere. Most of these data are displayed 
graphically in the following chapter. Due to the attrition in the study, the sample sizes 
vary as shown previously. The demographic data are based on all of the 
questionnaires that we received. Thus some pilots may appear in Q2 or Q3 or both, in 
these displays. All pilots appear in the Q1 data ~ filling out that form was the entry 
path into the study. 



Figure 111-2. Typical graphic representation of Likert scale results. 
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Confidentiality 


Volunteer pilots were assured of confidentiality. A method which was previously used 
in our studies was employed: for details, see Wiener (1989, p. 12). The confidentiality 
system was based on self-assigned combinations of letters and numerals to a 
maximum of six characters. In the portion of the Q1 form where the pilot signs up to 
join the study (Appendix D-2), pilots were instructed to assign themselves a code which 
they could remember, but would not identify them. They were also given a self- 
adhesive tag with a matrix of boxes into which they could enter their ID code; it was 
suggested that they keep it in a flight manual. The code was attached to the sign-up 
page. When we received the Q1 , we set up a separate computer file with the ID code 
and the pilot’s name and address, so that we could contact the pilot if need be. For 
example, we occasionally received a form with an entire page inadvertently left blank. 
No other record could link the pilot to his code, and this record has since been 
destroyed. Q2 and Q3 contained only the ID code, no names or addresses. 

Prior to recruiting pilot volunteers, we met with the Safety Committee of the IACP to 
outline the study and discuss the confidentiality plan. No concerns about 
confidentiality were raised by the IACP representatives, and they readily agreed to 
support the study. The investigators offered to brief IACP on the progress of the study, 
or to hold joint management-IACP briefings. Several of these meetings have been 
held, and cooperation with IACP was excellent. 

In other contacts with the pilots, confidentiality was also preserved by whatever means 
necessary. For example, in the face-to-face and telephone interviews (next section), 
we could not pretend that we did not know whom we were talking to. We simply 
explained this, and assured the pilot that we would not record any names or 
identification codes with the interview notes. The pilots were satisfied with this, in fact 
no question about confidentiality was ever raised. On jumpseat observations, no 
record was kept of crew names or flight numbers, dates, origins or destinations. We 
feel safe in saying that confidentiality was simply not an issue in this study. 


C. OTHER SOURCES OF INFORMATION 


Interviews 


Two sets of interviews were conducted. The first was at the initial period of the study, 
before 757 school had begun. The interviewees were flight management personnel, 
beginning with the vice-president for flight operations. Following him were flight 
standards pilots, the 757-767 fleet manager and assistant fleet manager, and others in 
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the management hierarchy. These interviews were face-to-face, and were conducted 
mostly one-on-one, with a few being one on two. 

The purpose of these interviews was to determine management attitudes toward 
automation, training methods, and standardization, and what problems they anticipated. 
The experimenter asked prepared questions, but allowed the interviewee ample room 
to discuss anything he wished. The information gleaned from the interviews was not 
treated as data, but as background material. 

The interviews with the management pilots yielded the following information: 

1 . A strong confidence in the choice of the 757-767 and the important role of these 
aircraft in the modernization of CAL’s fleet, and its route plans. 

2. A strong approval of the training plans and syllabi being drawn up by the fleet 
managers and their staff persons. 

3. Concern about safety problems in highly automated aircraft, and the ability of 
management to ensure, though training and other support, that automation 
would not be a problem. 

4. Concern about standardization in general, and the ability of flight management 
to standardize the 757-767 fleet to harmonize with the other fleets at 
Continental. The question of cross-fleet standardization and the desire not to 
make the automated aircraft “odd-balls” permeated every discussion. [These 
interviews were completed nearly three years before the company made the 
decision to buy the fleet of Boeing jets]. 

Jumpseat Observations 

The three authors, and one graduate student assigned to the project, made a number 
of jumpseat observations of line trips on the 757. This was for familiarization; no data 
were collected on these flights. 

Ground Schools 


Two of the authors attended ground school on the 757, including the program on 
Human-Centered Automation Training (H-CAT), and the international qualification 
class. Both authors had a CBI access code and worked on this instruction while in 
ground school. 



Standardization Meetings 


Two of the authors attended some of the 757 standardization committee meetings for 
instructional purposes, and at times were called on for advice on matters of checklists 
and procedures. The study team made several presentations on the progress of the 
study in standardization meetings. 
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IV. CHARACTERISTICS OF THE VOLUNTEER COHORT 


A. OVERVIEW 


In this chapter we provide in graphic and tabular form certain demographic information 
provided about themselves by the pilot volunteers. Most of the information deals with 
flying time and experience in various cockpits; a minor portion of the chapter covers 
such variables as the volunteers’ age, computer usage, and preferences for aircraft. 

The chapter is organized along the three longitudinal phases of the study and the 
questionnaire data collected at each phase. 

Representativeness of the Sample 

As we noted in a previous NASA report (Wiener, et al., 1991), an experimenter can 
never be certain that the sample of volunteers is truly representative of the overall 
population (all pilots in some circumscribed group). We asked ourselves the following 
question: are people who volunteer for a project, who are willing to give their time for no 
direct gain, attitudinally different from those who do not respond to the call for 
volunteers? This problem is known in statistical sampling as “non-response bias.” 

We have no ready answer to this question - the possibility of non-response bias 
plagues any study based on volunteers. It is generally supposed, but seldom proven, 
that volunteers for a study such as this may be the “sharper”, more capable, more 
motivated end of the continuum of aptitudes among the population from which they are 
drawn. If this is true, it may be extended to assume that the attitudes expressed are 
more positive toward fleet modernization and the role of automation. It might follow 
from this that the sample would contain proportionally more young pilots than the 
population, but this is speculation also. 

Why does a pilot volunteer to serve in such a study? We feel that we know the answer 
to this from interviews and open-ended questions. First, many are curious about the 
study, and many feel that volunteering for a study is the professional thing to do. 

Others may be attracted by the technological reputation and mystique of NASA. 

Finally, we feel that many, perhaps most, of the volunteers were drawn to the study by 
the persuasion of a popular fleet manager who personally called for volunteers at the 
first meeting of each new transition training class. 

We have not answered the question of non-response bias. We have found no obvious 
bias in our sample. We have every reason to believe that these responses can be 
generalized to the population of Continental pilots who bid the 757 in those years. This 
may present a bias in itself: who among the CAL pilots leaves a comfortable position in 
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a traditional aircraft, and for no monetary gain (see below) accepts the trouble and the 
risk of a transition to a modern aircraft? We have some answers to that question: in 
previous studies we asked volunteers why they bid the glass aircraft. The answer was 
usually framed in terms of seeing glass as the future of commercial aviation, and their 
desire to be on the leading edge of that trend. A number of the attitude probes (see 
page A-4) are aimed at pilots’ motivation to bid the new aircraft. 

Migration Patterns 

At the time of our study, Continental, unlike nearly every other airline in the U.S., did 
not pay pilots according to a “weight differential.” Pay was based on longevity at the 
company, and not on the weight of the aircraft flown. Longevity affected bidding for 
aircraft seats, trips, and bases, as it does at other airlines. This fact has implications 
throughout the company, as well as for this study. One of the things that the non- 
differential system does is to discourage “seat hopping," a practice which is costly for a 
training department, and generally wasteful of pilot talent. In other airlines, pilots bid 
from plane to plane in order to move up the path of weight differential and higher pay. 
Most pilot contracts attempt to limit seat hopping, due to its impact on training cost and 
flight proficiency. Some pilots seat hop in order to collect type ratings, as a hedge 
against possible job loss and re-entry into the market. 

At Continental, pilots tend to move around until they find a plane, a base, and a 
schedule that serves their needs, and stay put for what seems, by standards of other 
airlines, a long stand. The recent base closings and reductions, with Newark and 
Houston expanding, has shattered some pilots’ plans which were based on location 
rather than aircraft. With no weight differential, pilots based their bids on their own 
convenience, and did not particularly care which aircraft they flew. Bids reflected 
desires for bases, schedules, long periods off, etc., and not so much for aircraft type. 

Non-differential pay schedules, though financially efficient for the company, can lead to 
some peculiar results, e g. senior captains flying low-end aircraft (DC-9, B-737-200), 
while junior captains were flying DC-1 Os over the Pacific. We once rode jumpseat in a 
DC-9 with a captain who was one of the most senior in the airline. With his seniority, 
why was he flying a DC-9, when “heavy metal” (DC-1 Os and B-747s) and international 
schedules were available to him? His reply was that he was tired of hotel living, and 
with his seniority could consistently bid out-and-back trips from Houston (IAH). He 
boasted that in the last five years he had spent only one or two nights a month in a 
hotel. 

We found in our early interviews that one of the motivators for a 757 bid was the 
anticipated fleet of 767s, which was on order for ETOPS operations across the North 
Atlantic, a highly desirable route. This turned out to have its negative side for this 
study. When, for economic reasons, the company decided to cancel its 767 order, 
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there was widespread dissatisfaction among the 757 pilots, due to both the reduction of 
the ETOPs flights, and the 757 schedule in general. At Continental the 757 and 767 
are in the same fleet. Because the company could not make good on the 767 lines, 
they agreed to waive the two-year rule (training freeze) and allow the crews to bid off of 
the 757. This resulted in a serious loss of pilots in our cohort, one of the hazards of 
longitudinal studies, as noted previously. The 757 program proceeded, and a new fleet 
of 777s and 767s are now being delivered. 

Thus, a combination of the loss of the 767 fleet, the generally undesirable schedules 
(in terms of the effort required to earn flight hours), and the assignment of more and 
more 757 time to Newark (EWR) and less to IAH led to heavy out-migration from the 
program in the first year of our field study. 757 time scheduled at the Guam base 
remained unchanged. The situation has now stabilized, and the promise of the 767s, 
with first deliveries scheduled for 1999, has once again made this a desirable fleet. 
[Deliveries of 26 767-400s will begin in July 2000], 

In June 1998, after data collection on this study had been completed, a new contract 
changed all of this, and put Continental pilots on a traditional seniority and weight 
differential basis. The entire airline had a “flush bid”, meaning that every pilot could bid 
any plane in the fleet. The bidding at CAL was simplified somewhat by a contract that 
created only three weight classes: 

Narrow body DC-9, 737-300/500/700 

Mid-body 727, MD-80, 737-800, 757 

Wide body DC-10,767,777 


B. PHASE 1 DATA 


Repeating what was said previously, the first questionnaire (Q1 ) was attached to the 
invitation to join the study, To sign up, the pilot filled out the questionnaire, which 
included the questions whose results are presented in this chapter, as well as a 20-item 
Likert attitude scale. There was one page that dealt with the confidentiality and self- 
assignment of the ID code, and an informed consent sign off. The completed forms 
were then sent to us at NASA Ames Research Center in an addressed, stamped 
envelope provided with the invitation and questionnaire. 

One attitude probe which appeared on Q1 that was not appropriate in the later phases: 
No. 8a, “I am very apprehensive about going through this transition” (see Figure 111-2, 
previous chapter). The results of the Likert probes are shown in Appendix A. 
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Miscellaneous Information 


Of the 101 volunteer pilots who submitted all three questionnaires, all but one were 
males. One other female pilot filled out Q1 , but we never received data from her again. 
Accordingly, we use principally the male gender in this report. Volunteers were asked 
to give their age to the closest month. We converted this to years for graphic purposes 
(see bar chart below). The age distribution is consistent with what we have seen in 
other field studies: the 757 tends to be a mid-career choice. At other airlines with 
weight differentials in their contracts, bidding the 757 represented something of a 
stepping stone, from lighter (and therefore less lucrative) aircraft such as the B-737 and 
DC-9/MD-80 to the “heavy metal.” At CAL, with no weight differential, and various 
reasons for bidding the 757, we still see mid-career pilots making this transition. 



FIGURE IV-1 . Age of pilot volunteers. 


We were astonished at the number of “older” (with respect to a mandatory retirement at 
age 60) pilots bidding the 757(see Figure IV-1 above). About 1/3 of the pilots were 
within ten years of retirement when they filled out Q1. With nothing to gain monetarily, 
this bid probably represented a desire to fly a modern aircraft before retirement. Many 
reported to us that even with only five to ten years left, they saw the company rapidly 
expanding with a glass fleet, and wanted to be part of that movement. We will look y at 
where (what aircraft) these pilots arrived from. Professional pride played a big part in 
the bidding. 

We have said little so far about first officers. Much of what we have already said of 
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captains applies as well to the first officers, and as we will see, their attitudes are 
remarkably similar. Our first officers tended to be mid-career in that seat, and 
jockeying for position as a captain of a smaller aircraft. Some senior first officers 
remained in the right seat of the 757, awaiting an opportunity to bid 757 captain. For 
most first officers, unless they are very senior, a more inviting route is to bid captain in 
the expanding fleet of B-737-500s and next generation 737s. Here they will find the 
opportunity to utilize glass cockpit knowledge and skills learned on the 757. As 
seniority builds, there is always a future in the expanding 757/767 fleet. 

The present base to which the volunteer was assigned, and his expected post- 
transition base, were also asked on Q1. The present base is displayed in the pie chart, 
Figure IV-2. The anticipated future bases are easily summarized: 


Base 

No. 

% 

Houston 

64 

43.5 

Newark 

78 

53.1 

Other 

5 

3.4 


Note that these are pilots’ expectations; the reality of assignments is where the 
company chose to base its 757 fleet. As it turned out, these were fairly realistic 
estimates in the aggregate. One can easily see the influence of Continental’s two 
primary bases, Houston and Newark. During the course of the study there was 
considerable base realignment and closing, including the once powerful and highly 
desired Denver base. Presently 757s are based only at Houston, Guam, and Newark. 
This report essentially ignores the Guam base, which even now is a small, somewhat 
remote part of the 757 operation. The concentration of 757s at Newark has turned 
Continental 757 pilots into a tribe of commuters, many continuing to live near formerly 
thriving bases such as Denver and Los Angeles, locations of reduced importance at 
Continental, but still considered desirable places to live. 
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Present Base 



FIGURE IV-2. Base of volunteers at time of 757 
transition. 


Computer Experience 

In a previous study (Wiener, et at., 1991) the authors were curious about the pilot’s 
computer experience, and hoped to relate this to performance in the cockpit. 
Unfortunately we asked the question the wrong way, asking merely was there a 
computer in the home? We should have asked who used it. We sought to remedy that 
problem in this questionnaire by asking if there was a personal computer in his home, 
and if so, how often did he (the pilot) use it? For the exact format, see page D-4, 
questions 1 0 and 1 1 . 

The answer to the first question (see Table IV-1 ), “do you use a personal computer in 
your home,” yielded the following results: 


Yes 

76% 

No 

23% 


It is often assumed that first officers, being younger than captains and therefore 
educated in the era of digital computers, are more likely to be intrigued by and 
competent in computers. One might expect to find a higher proportion of first officers 
with home computers. This is argumentative: one could advance the position that 
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captains can more easily afford computers. First officers may not have the cash to put 
into computers. We examined this by casting the data into a 2-by-2 table and 
performing a chi-square test of row/column independence. This resulted in a chi- 
square of 1.03, (df = 1 ), indicating that the difference in home computer usage between 
captains and first officers (80 % vs. 73 %) was not statistically significant. 

For the roughly 3/4 of the sample that responded positively, the breakdown by level of 
usage is given in Figure IV-3. Well over half of the sample reported usage daily or 
several times weekly. We again examined computer usage by captains and first 
officers, casting the data into a 2-by-4 matrix (Table IV-la). Again the result was again 
non-significant: chi-square = 2.04 (df = 3). 

In summary we find that in our sample of 146 pilots, about 3/4 report having and using 
a personal computer, with fairly uniform distribution of cases over the four levels of 
usage. We find no difference between captains and first officers in the availability or 
usage of the home computer. We shall next attempt to correlate these data with 
attitude items. We are unable to do what we had originally planned (using the training 
books), attempt to correlate computer usage with proficiency measures. 
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Personal Computer Use 



FIGURE IV-3. Frequency of usage of home computer. 


DO YOU USE A PC AT HOME? 



TABLE IV-1. Computer usage by seat. 
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PERSONAL COMPUTER USAGE BY SEAT 
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146 


TABLE IV-la. Frequency of usage of home computer by seat in 757. 


In Table IV-la we have displayed frequency of computer usage by the crew members 
as a 2-by-4 contingency table. We will discuss the outcome of statistical tests on this 
and other tables in the next chapter. 

The only other experimenter that we are aware of who has gather statistics on 
computer usage in high tech crew members is Orlady (1991), who asked pilots of high- 
tech cockpits how many had home computers. The group was about evenly split. 

Note that if his report is taken literally, he made the same mistake as Wiener et al. did 
in their 1991 study. The proper question is not ownership, but usage. Orlady took it a 
step further and asked the group that responded thatthey did have computers whether 
they felt their computer experience made any difference in transition to glass. The 
group was about evenly split (Orlady, 1991, p. 2.12). 
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Choice of Aircraft 


We asked a question we have asked before, requiring the volunteer to pick from his 
company’s fleet the plane that he would most want to fly, quality of trips and pay being 
equal (see page D-4, question 14, for exact wording). The results are displayed in 
Figure IV-4. The results indicate a strong loyalty for the 757, accounting for nearly 3/4 
of the votes. The DC-10 and the B-747 accounted for most of the rest, the remaining 
aircraft, narrow bodies with one exception, obtained few votes. 


First Choice of Aircraft Type 
Regardless of Money, Base, Etc. 



FIGURE IV-4. Pilots’ choice of aircraft in Continental 
fleet. 


Total Flying Time 

The two bar charts showing total flying time and flying time at Continental reinforce 
what we have said about the mid-career pilot, those in a range of perhaps 10,000- 
16,000 hours. We have made a distinction in the two flying time charts between total 
time and time at Continental. These disparities exist because, as explained early in 
this report, Continental is not a “pure bred" airline, but one composed of many 
tributaries (see Chapter I). A large number of the pilots came to Continental in recent 
years, as a result of the airline mergers and acquisitions engineered during the reign of 
Frank Lorenzo. Thus the difference between the hours of flying time depicted in the 
two figures (IV-5 and IV-6) is considerable. 
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FIGURE IV-5. Total flying time, all sources. 



FIGURE IV-6. Total flying time at Continental. 
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The typical Continental pilot in the 757 program, at the time we collected data (mid- 
1990s) had about 8,000 to 13,000 total hours, including pre-merger companies, 
military, general aviation, etc., about half of which is with Continental. Like the pilots 
whom we have studied in other projects (Wiener, 1989; Wiener et al. , 1991 ), the B-727 
predominates. At every airline we have studied, this is the case. We call the 727 the 
“prep school for 757.” Table IV-2 illustrates the importance of the 727 in the migration 
patterns of the 757 pilot. 

Transition from the 727 (or the DC-9 for that matter) to the 757 is a turning point in a 
pilot’s career: a sweeping technological change, and a challenging training program. 

At most of the airlines we have studied, we have encountered the “25-year 727 pilot.” 
Every airline has a collection of them. He (or she) has spent an entire career in the 
three seats of the 727, and has little interest in moving. What it takes is a new 
technology aircraft, not just a heavier one, and perhaps a subtle threat that the 727 is 
going to soon be retired. One thing that makes the 25-year 727 pilot somewhat 
apprehensive about bidding the 757 is that he has been to school so little during his 
career on the 727, compared to pilots who have migrated all over the fleet. And at 
Continental, with no weight differential, why bother? 

Previous Cockpit Positions 

Pilots were asked to fill in a matrix similar to Table IV-2, simply checking each cockpit 
position they had held at Continental. They were instructed not to include flying time in 
each seat, only a check that they had at least once held this seat at Continental. Some 
interesting facts come from this table. We again observe the importance of the 727 in 
the migration of pilots. Pilots came from variety of seats, including captains who had 
flown the three wide body (“heavy”) jets that were in operation by Continental when the 
study began (A-300, B-747, DC-10). The A-300 has since been retired. Again turning 
to our discussion of the lack of weight differential pay scales at Continental, it is 
probable that such a bid would not have occurred at other U.S. airlines. The 757 is a 
mid-weight aircraft, somewhat heavier than the other narrow bodies, far lighter than the 
wide bodies. At an airline with weight differentials, it would be a significant financial 
sacrifice for a wide body pilot to bid the 757, whatever his motive. 
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Previous Seats At Continental 

Aircraft Type 

Captain 

First Officer 

Second 

Officer 

Totals 

DC -9 

25 

38 


63 

MD-80 

15 

17 


32 

737-100/200 

9 

6 


15 

737 - 300 

23 

22 


45 

727 

52 

64 

57 

173 

A-300 

21 

9 

2 

32 

707/720 

4 

19 

22 

45 

DC -10 

26 

37 

23 

86 

747 

14 

14 

6 

34 


Table IV-2. Previous seats held at Continental by pilots in the cohort. 


We next asked the pilots what was their last aircraft before embarking on their 757 
transition. Figure IV-7 shows the results. It is noteworthy that they came from so many 
aircraft, with a sizeable number coming from the three wide bodies. We have not 
attempted to scale these results to the number of aircraft (and crews) in each fleet. The 
DC-10, A-300, 737, and 727 are about equal in their contribution, and the four account 
for almost 3/4 of the pilots in the cohort. At the time the first 757 classes were being 
formed, the A-300 was on the way out of the company’s fleet; their pilots were 
scrambling for the best deal they could find. The 737-300 at the time was the 
company’s most modern cockpit. Continental’s models had the flight management 
system (FMS), but not the glass cockpit. This configuration is often called the B-737- 
300-non-EFIS. At this time the fleet of glass B-737-500s began arriving. 
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FIGURE IV-7. Last aircraft flown before 757 transition. 



FIGURE IV-8. Months in last seat before 757 transition. 
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For the 737-300 pilots, familiar as they were with FMS functionality and Boeing 
terminology, this was a relatively easy transition. 

In order to determine the stability of assignment of the pilots who bid the 757, we asked 
the number of months in the model flown before transition. These data are displayed in 
Figure IV-8. Examination of the figure shows that a sizeable group (27 per cent) 
resided in the left-most bar (zero to 24 months). Following them were three roughly 
equal subgroups (25-96 months) accounting for about 55 per cent of the sample. Four 
small subgroups of those with a large number of years accounted for less that 20 per 
cent of the total. 

How many of the transitioning 757 pilots had previous glass experience at Continental 
or elsewhere? Pilots were asked merely to list any glass cockpit of any type in which 
they had been assigned. Table IV-3 displays the results. About 3/4 were 737-500, the 
rest being a scattering of military and executive jet cockpits. Thus, if we exclude for the 
moment previous 737-500 crews, it can be said that the vast majority in the cohort had 
not been exposed to glass prior to 757 transition. 

Glass or not, we wished to know the most advanced cockpit which the pilot had flown at 
any time during his career. The exact meaning of “most advanced” was left to the pilots’ 
discretion. These data are shown in Table IV-4. The table indicates a wide variety of 
airline, military, and executive jet cockpits in the experience of the cohort. There are 
some discrepancies between these data and the previous question about glass 
experience, probably due to misunderstanding the question. 
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Previous Glass Experience 

Aircraft Type 

Responses 

F-14-A 

1 

737 - 500 

27 

G - IV 

2 

Beechcraft C - 90B 

1 

F - 16 

1 

747 - 400 

1 

EMB-120 

3 

FJ50/LR55 

1 

737-500/MU-300 

1 

Saab 340 

1 

ATR 42 

1 

Total 

40 


Table IV-3. Glass aircraft flown by cohort 
prior to 757 transition. 


For example, we cannot explain the fact that ten pilots claimed to have 757 experience, 
but this is not reflected in the previous question about glass experience. Also there is a 
minor discrepancy: 31 pilots claim 737-500 experience, but only 27 listed the -500 in 
response to the question about past glass cockpits. With the rapid fleet expansion at 
Continental, with large orders from Boeing for 757, 767, 777, 737-500, and recently for 
next generation 737s, and the retirement of the older model aircraft, the figures will 
change dramatically in the next five years. If these questions were asked five years 
from now, undoubtedly most pilots would have glass experience, most would list some 
new model of Boeing aircraft as their most advanced cockpit, and they would not be 
going through glass transition for the first time. Early in the next century Continental’s 
fleet will be all Boeing and all glass. 
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Most Advanced Aircraft 

Aircraft Type 

Total 

727 

6 

737 - 200 

1 

737 - 300 

17 

737 - 500 

31 

747 

4 

757 

10 

A300 

12 

DC -10 

20 

MD-80 

21 

Saab 340 

1 

ATR 42 

1 

F - 4 

2 

F - 15 

1 

F - 16 

3 

FJ50 

1 

C -141 

2 

53 -A Viking 

1 

Westwind 1124 

1 

G lil 

1 

G IV 

1 I 


Seven responses each involving two aircraft were not included in the table: 

1 . DC- 1 0/A-300 5. 757/767 

2. DC-10/L-101 1 6. 757/767 

3. DC- 1 0/C- 141 7 . DC-9/727 

4. DC- 10/737-300 


TABLE IV-4. The most advanced aircraft flown prior to 757 transition. 
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Recent Experience with Various Instrument Approaches 


In past field studies we have asked the volunteer pilots to estimate how many times in 
the previous 12 months they have flown, either as pilot flying (PF) or pilot not-flying 
(PNF), various instrument approaches. [Currently at CAL, the PNF is called the “pilot 
monitoring” (PM)]. The results are displayed in the next five figures. The data must be 
regarded as estimates, based on the pilots’ memory of the previous year. These data 
have been deemed valuable in planning training for the less-frequently used 
approaches. Such data may be particularly useful in planning training syllabi and 
schedules for the Advanced Qualification Program (AQP). 

The question sometimes arises during an accident investigation, when the pilots fly an 
infrequently used approach and an accident results. Such was the case in the crash of 
the U.S. Air Force B-737 (T-43A) in Croatia (Phillips, 1996a). It is not unusual in these 
cases to find that the pilot has made few, if any, of the less-often employed non- 
precision approaches in the last 12 months. 

The data on autolands present a special case. Some of the pilots would have spent 
the last year flying older aircraft not configured for autoland, or possibly configured but 
not maintained for autoland (see Fig. IV-7). Be that as it may, the frequency of 
autolands in the pre-1994 Continental line experience of the early 757 cohort was 
virtually zero. Close to 90 per cent of our volunteers reported no autolands, and the 
remaining frequencies are minimal (see Fig. IV-9). The following Continental aircraft 
were equipped authorized for autoland at the time of our study: B-737-300/500 and B- 
757. The MD-80 was equipped but not used. 



FIGURE IV-9. Frequency of autolands in previous 
year. 
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FIGURE IV-10. Frequency of ADF (NDB) approaches. 


Figures IV-10, 1 1 , and 12 demonstrate that the frequency of various non-precision 
approaches is quite small. About 55 per cent of the pilots report no ADF (more 
correctly, NDB) approaches during the previous year, and the frequency is very low for 
the remaining pilots. The frequency of localizer and VOR approaches is also quite low: 
about 30 per cent of the pilots report having flown none of these two approaches during 
the previous year. 

We were somewhat surprised by the low usage of Category II ILS approaches. About 
45 per cent of the pilots reported zero Cat II approaches. All of CAL’s fleets were 
qualified for Cat II. The following were qualified for Cat III at the time of the study: 737- 
500, 757, MD-80, and 737-300. Now the 777 and 737NG can be added to the list, and 
the MD-80 is Cat III qualified. 
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FIGURE IV-1 1 . Frequency of VOR approaches. 
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FIGURE IV-13. 
approaches. 


Frequency of Category-ll ILS 




C. PHASE 2 DATA 


In this section we shall discuss the demographic data of Phase 2. This phase was 
timed to be after the end of training and initial operating experience. Volunteer pilots 
were sent Q2 forms approximately 3-4 months after they entered the study. This period 
allowed time for transition training (ground school and simulator), IOE, vacation time, 
and at worst about two months back in their previous seat while awaiting a 757 seat 
(see Figure IV-14 and 15). 

As the study moved on into the latter half of the 1990's, this became less of a factor. 
Pilots went straight through the program and joined the line without interruption. 

Thus, if our timing was right, and if the pilots filled out the Q2 form promptly, one may 
think of the second phase questionnaire as being close to the initial point of a pilot's 
line experience. His formal training was complete, and he would now be learning 
through on-the-job training. We should also note that this is the point, early in our 
study, at which our sample size diminished, due to the bid-off of the 757, due largely to 
what were perceived as poor schedules. 

Post-Training Assignment 

In the first year of this study many pilots completed training, and in some cases IOE, 
and then had to be assigned to their former aircraft for typically two months until a 757 
seat was available. One question simply asked if they were assigned after training to a 
757, or their former plane. The results of this question are tabulated below. 


757: 84 % 

Former plane: 16% 

The cockpit they returned to is summarized in Figure IV-14. 
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Aircraft Assignment After Training 



FIGURE IV-14. Aircraft assigned to following 757 
transition 


For those who did not go the 757 immediately, the number of months of reassignment 
to their former plane, before moving to the 757, is shown graphically in Figure IV-15. 
This was in part due to the difficulty of balancing training through-put with new aircraft 
arrivals, during a period of rapid fleet expansion. No fleet manager wants to get caught 
short of flight crews with new aircraft on the tarmac. The obvious solution is to absorb 
some costs and train to inventory, hoping to make use of the excess pilots in their old 
plane until a 757 billet is available. They did not do badly: Figure IV-14 and the data 
on the previous page show that the vast majority of pilots went directly from 757 
training to the 757 line, and the maximum delay for those who did not was three 
months. The following ASRS report addresses this issue. 


NARRATIVE. Finished checkout F/O 6/89. No position until 10/89. Flew simulator in 9/89 
for 90-day landing currency. You could say the find points of working the FMC had escaped my 
memory. We were cruising at FL390 and received clearance to FL410. Captain loaded in MCP 
glare shield altitude— at which point I asked how he input the data for the climb, neither 
monitoring to confirm the climb to FL410. Several mins later Center asked if we had climbed. 
"No, still at 390." The altitude had not been put in the FMC, and we were navigating with VNAV 
and LNAV. Both crew members low experience levels in type contributory to the altitude 
oversight. Factors affecting performance: 1) supervision management practice of putting 2 
inexperienced crew members together, or 2) just not monitoring/keeping track of crews' levels of 
experience; and 3) after training crew member on advanced/automated cockpit, waiting an 
extended period before assignment to aircraft. Fly the aircraft. ACCESSION NUMBER: 

124912 
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Months Before Returning to 757 



FIGURE IV-1 5. Months following 757 transition 
training before being assigned to the 757. Zero months 
represents those going directly to 757 following 
training. 


Flvino Experience 

Pilots were asked to estimate their total 757 flying time. These data are displayed in 
Figure IV-1 6. It was expected that at this point the 757 time would be quite low. We 
estimated that the pilot would have, at best, 200-300 hours. Some pilots had as much 
as 500 hours. The very high times shown in Figure IV-1 6 can only be from pilots who 
did not send the form back promptly, and amassed flying time before filling out at least 
that question. This graphic will be displayed again in this chapter, when data from P3 
are presented, in order that it may be compared to the flying time of the pilots in P3 
(over a year later). 

On the following page we have two bar charts showing 757 flying time at Phase 2 and 
Phase 3. Note that the plots are on different scales. On the top graph (Phase 2), the 
bars are 1 00 hours apart; on the lower they are 200 hours apart. 
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Total Flying Time in 757 



Hours 


FIGURE IV-16. Total flying time in 757, at Phase 2. 



FIGURE IV-17. Total 757 flying time, at Phase 3. 
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D. PHASE 3 DATA 


The third and final phase of the experiment (P3) was designed to be approximately one 
year after IOE. In other field studies that we have conducted, this time is usually found 
to be a turning point at which the pilot starts to “feel comfortable.” Although there is 
always more to be learned, at this time, with a year’s line experience behind him, the 
pilot new to the glass cockpit has mastered the FMS functionality, autopilot modes, 
display modes, etc., and has probably also mastered the “tricks” of line flying a glass 
aircraft. 

Comfort with the Aircraft 

In this study the pilots appeared to “feel comfortable” (a phrase widely used by pilots) 
much earlier than our previous work would forecast. Figure IV-18 shows the result of 
the compound question in Q3, “Do you feel ‘comfortable’ in the 757 now? (Y/N). If yes, 
how long after you went on the line did it take (months)?” As to the first question, 97 % 
reported “yes”, they felt comfortable. The durations on the line are shown in Figure IV- 
18. Almost half of the respondents reported two months or fewer, and a very small per 
cent offered replies of over six months. Such favorable results are probably due to the 
user-friendliness of the training program. The high confidence and high regard that this 
cohort had for the transition training program emerges in many places in this report: 
e.g. attitude probes, interviews, and the open-ended questions in Q2 and Q3. 


How long did it take you 

to feel comfortable (months)? 




FIGURE IV-18. Months (following training) before pilot 
felt “comfortable” flying the 757 in line operations. 
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Flvina Time. Phase 2 and Phase 3 

The total hours of 757 flying time at Phase 2 (actually whenever the respondent filled 
out his questionnaire) is depicted in Figure IV-16. Figure IV-17, the 757 time at Phase 
3 is displayed on the same page for comparison. Phase 2 was designed to be 
approximately 3-4 months after transition training. Again, the calculation of the 
intervals between phases is at the mercy of the pilot volunteer and when he fills out this 
questionnaire. For PI and P2, we tailored the delivery of Q2 and Q3 to each pilot, 
attempting to deliver the questionnaires to him based on the nominal time of the 
phases. Viewing the two figures together allows one to see the growth in 757 flying 
time during the (nominal) year, from IOE to the point at which he filled out the Q3 form. 

In the following narrative, we see a captain who has only been on the aircraft one 
month who handles a very difficult emergency, and attributes his success to the training 
he received, as well as the competence of the first officer. We believe the reader will 
agree that this is a good example of airmanship by a captain with an extremely low 
time-in-type. Note that it is obvious that this report is from an Airbus, as the term 
ECAM is used. 

NARRATIVE. Aircraft was in Cruise at FL290 due to the prior shutting down of the 
number-two pack for overhead. We were just preparing to descend to cross Bradford 
intersection at FL240 when we heard a possible compressor stall and the aircraft shook 
and yawed. We got confirmation of the number-two engine failure in the electronic 
controlled aircraft monitoring ECAM [an Airbus term which is the counterpart of 
Boeing’s EICAS], ECAM procedures were followed. The first-officer was flying and I 
allowed him to continue to do so. I did not elect to do a restart as the EGT was climbing 
rapidly. I shut down the engine according to ECAM and used the fire bottle due to high 
and rising EGT. Started APU, declared emergency with ATC, notified Dispatch, made 
public-announcement to passenger, subsequently lost APU before landing, wouldn't 
restart. Elected not to prepare cabin for evacuation and weather was VFR. Landed 
without incident at Chicago and taxied back to gate as all hydraulic systems were 
operating normally. The number-two engine compressor section had failed completely 
and broken up. The engine was replaced by maintenance. I was fortunate to have along a 
very competent first-officer, and although I had only been on the aircraft a month, training 
had prepared me very well to handle the problem. ACCESSION NUMBER: 284470 
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V. ANALYSIS OF QUESTIONNAIRE DATA 


A. METHODS OF ANALYSIS 


Because most of the variables examined in this study were categorical (e.g. captain 
versus first officer; or previous glass experience versus none), the data collected are 
best suited to nonparametric analysis. Where possible these methods were employed, 
and interval data (such as age and flying hours) were divided into categories (see 
Chapter IV). In this chapter we shall report and comment on those data that were 
analyzed and subjected to statistical tests. Much of the data are reported in Chapter IV 
as merely descriptive data, not suited to statistical analysis (e.g. choice of favorite 
aircraft in company’s fleet). In some cases, for statistical convenience, the data are 
treated as being on an interval scale, when more correctly they are on an ordinal scale. 
For example, the intercorrelation matrices were computed using the Pearson product 
moment method, which properly requires interval data, but is widely used for ordinal 
data, such as responses on a Likert scale. 

Other data were subjected to cross-tabulation tests (contingency tables) using the chi- 
square distribution. An example would be determining if there is a relationship between 
a variable which we have called “SEAT’ (captain vs. first officer) and some other 
categorical variable such as computer usage (yes/no). Unfortunately our sample size 
is small for the number of variables examined, and some compromises with statistical 
purity were made. In the case of contingency tables, there were often low frequencies 
at the extreme points (corners of the matrix), so the results may be inexact. 

In the case of the intercorrelations of the Likert variables, each pair taken together 
potentially produced a 5-by-5 matrix. In some cases there were no responses at all for 
a given response category, thereby reducing the matrix. Usually those cases involved 
one or both of the two extremes where the respondent could “ strongly agree” or 
“strongly disagree”. We have seen in this study, and in previous ones, a tendency on 
the part of the pilot volunteers to avoid the extremes, for example, the graphs on page 
A-1 1 . No pilot responded in the “strongly disagree” category in response to that probe. 
There are no cases in our data where there were fewer than four non-zero categories. 
Eleven tests involved reduced matrices due to one extreme (either “strongly agree” or 
“strongly disagree”) yielding no responses. 

Throughout this report we use the conventional value of .05 for the statistical 
significance level (alpha, or probability of a type-1 error). In interpreting the correlation 
matrices, an absolute value of Pearson’s r > .163 is significant for n = 147 for two-tailed 
null hypotheses. For n = 146, where the data from one volunteer may be missing, the 
critical value is very slightly higher in the third decimal place. 
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B. PHASE 1 


Intercorrelations 


For each of the three questionnaires (Q1, Q2, and Q3), an intercorrelation matrix of the 
responses to the Likert probes was computed as described above. Additionally ten 
selected demographic variables were included in the original matrix, but are not 
included in the matrices reported here, due to the fact that their inclusion would result 
in a vastly expanded matrix. A copy of any of the entire intercorrelation matrices 
mentioned in this report is available to the qualified requester. 


The size of the entire square, symmetric intercorrelation 
matrix is a function of the square of the number of variables 
included; a single echelon of the symmetric matrix would 
contain, for N variables, N(N-1)/2 correlations. If the 
computer program prints the entire square matrix, there are 
N 2 correlations. 


The Q1 questionnaire included 20 Likert scale probes, resulting in 190 correlations. 

Had we included the ten demographic variables, there would be 465 correlations. With 
our statistical software, the resulting matrix would require 18 printed pages. 

Accordingly the correlations between the demographic variables and the Likerts were 
examined separately. Some of the terms were obvious in their correlation and hence 
are not reported, for example the positive correlations between total flying time and 
flying time at Continental, or some of the obvious correlations of variables (e.g. AGE, or 
total flying time -- TFTALL) with “SEAT" (captain vs. first officer). We chose from the 
list of 1 1 , six variables to be included in the correlation matrix for Q1 . There were 
originally ten variables. We later derived GLASSNU (see next page). The variables are 
described on the following page. 
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AGE 


SEAT 

PCUSE 

TFTALL 

TFTCAL 

GLASSNU 


pilot’s age (months) 

position on 757 (“1" = Capt., “2" = F/O) 

does pilot use PC at home at all (“I” = yes, “2" = no) 

total flying time, Continental and elsewhere (hours) 

total flying time at Continental (hours) 

had the pilot, previous to 757 transition, flown any glass cockpit? 
(“0" = no, "I" = yes) 


The following list are demographic variables not included in the analysis. See 
Chapter IV for descriptive statistics of these variables. 


CHOICE 

LASTACFT 

LASTMOS 

LASTSEAT 


Pilot’s first choice of aircraft in company’s fleet 
Last aircraft flown before 757 transition 
Months on last aircraft before 757 transition 
Seat occupied before 757 transition (Capt. vs. F/O) 


TABLE V-1. Demographic variables on Questionnaire 1. 


We shall now examine some of the variables and their relationships. 

Pilots' Aae 

There has been considerable interest in both the airline community and the research 
community on the effect of chronological age of the pilot going through his first glass 
transition. Much of what was said was based on unsupported, usually negative 
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stereotypes of older workers — that the senior captains were not abreast of technology, 
since they did not grow up in the computer age, and that they were mentally 
unadaptable to the high tech cockpit. We do not know of any research on the topic of 
age and transition to glass. There has been considerable interest in pilot age in the last 
three decades due to the legal requirement that pilots flying under FAR 121 retire at 
age 60 (the “age 60 rule”). But the research to support that rule pre-dated the era of 
the FMS cockpits. 



FIGURE V-1. Example of generally accepted probe 
which correlated with age of pilot. 


Training personnel spoke of apprehension borne by older pilots. In our interviews at 
other airlines with pilots in glass transition for the first time, there was frequent 
expression of apprehension concerning the demands of the transition training, 
particularly with respect to their lack of computer skills. These concerns always seem 
to come from captains: investigators did not encounter, in previous studies, 
apprehension on the part of the first officers making the transition. In our interviews 
with pilots and instructors in the current study, we have heard less of this. The graph 
(8A) from Questionnaire 1 may be instructive. It is shown as Figure 11-1 . Only about 
10% of the respondents expressed agreement with the probe. We see in this graph a 
rather strong rejection of the opinion that pilots arrive at their transition training filled 
with apprehension. This subject is also discussed in Chapter II. 

Chi-square tests of the 20 contingency tables of attitude response vs. seat (captain and 
F/O) all resulted in negative findings. We have found no difference between captains 
and first officers in replies to this or any other attitude probes, in any of the three 
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questionnaires, or on questions about computer usage. In brief, we have found no 
differences between captains and first officers in the 757 program, except obvious 
factors such as age and flying experience. Therefore in further analyses we have 
aggregated the data of the two positions into one. 

Our inquiries in previous studies (Wiener, 1989) into the influence of age revealed that 
if there is any reliable generalization, it is that the older pilots seemed to get off to a 
slow start in early days of ground school, having a slight amount of trouble mastering 
some of the new concepts, compared to the younger first officers, who were presumed 
to be “techies”, skilled in digital concepts and operations. After this initial period, the 
captains, drawing on their vast experience and airmanship, caught up and by the time 
they reached the simulator, were performing at a high level. It was unfortunate that the 
training book data did not work out. They might have provided somewhat objective 
information on the effect of age during training. 

AGE correlated significantly with one Likert probe, No. A12, “I have no trouble staying 
‘ahead of the plane'”. (See Figure V-1 , previous page.) Since the Likert scale, when 
treated as an interval scale, goes from 1 to 5 as it goes from strongly agree to strongly 
disagree, a negative correlation means that high age goes with low Likert values 
(approval of the probe). In this case, the older the pilot, the more approving he is of the 
statement that he can easily stay ahead of the plane. The younger pilots may have 
some reservations about their own abilities. 

For obvious reasons the variable SEAT (captain vs. F/O) is highly (negatively) 
correlated with AGE (r = -.54). In SEAT, as we have indicated, captains are coded as 
“1"s and first officers as “2"s. The lower index number (captains) is associated with 
higher age. For this reason, SEAT also is positively correlated with A12. 

In summary, we have not produced any evidence on differences due to the trainees’ 
age. One significant Likert, and somewhat obvious correlations, is the best we can 
offer. The age question will have to await perhaps a simulator study in which 
dependent variables can be carefully measured and examined with respect to the 
pilots’ ages. 

Seat 

The variable SEAT refers to the seat that was bid for 757 training, “1" for captain. In 
most cases the seat in the 757 bid was the same as that held at the time of bid. A few 
senior first officers bid for 757 captain seats, and made the transition and upgrade at 
the same time. SEAT is highly (negatively) correlated with AGE (r = -.54) due to 
seniority considerations. SEAT is also correlated with A12 (r = .17), possibly through 
its correlation with AGE. It also correlates (r = .18) with A10 (“I am not concerned 
about making errors, as long as we follow procedures and checklists”). This indicates 
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that the captains are more accepting of the probe than the first officers. This finding, 
and the one indicating a positive correlation between age and A12, suggest a degree of 
caution and conservatism on the part of the first officers, and self-confidence on the 
part of the captains. This runs counter to the popularly held stereotype of the ultra- 
conservative captain. 

The relationship between seat and attitude was also tested by forming a 2-by-5 
contingency table (2-by-4 in those cases where an extreme [SA or SD] had zero 
entries), with attitude choice as a column variable and seat as a row variable. The chi- 
square contingency coefficient was computed and tested for all 20 probes. None 
resulted in a rejection of the null hypothesis of row/column independence. 

Flvina Experience 

The two measures of flying time, TFTALL and TFTCAL naturally correlate highly with 
each other (r = .81 ). This correlation is obvious, since the pilot’s total flying time, 
TFTALL, contains the value of the variable TFTCAL, his flying time at Continental. 

They also correlate, as one would expect, with SEAT and AGE. 

TFTALL correlates (r = .16) with A18 (“Continental’s CRM training has been helpful to 
me”), positive correlation indicating that pilots with high flying hours tend to take a less 
favorable view of the CRM training. This is probably due to the correlation with rank: it 
would indicate that low-time pilots (mostly first officers) are more accepting of CRM 
than captains, which is the experience at most airlines. It is interesting, and not easily 
explained, that the correlation of TFTCAL with this probe was very small. Total flying 
time appears not to be a particularly fruitful variable. In the discussion to follow, of 
Questionnaire 2 and 3, the interest will shift to 757 flying time as a predictor variable. 

GLASSNU was a derived variable, based on the question about prior glass experience. 
We created this variable in order to test hypotheses about attitudes as a function of 
having flown or not flown glass aircraft before 757 transition. The variable we created 
was a “(0, 1 )” variable: it recorded only yes (1 ) or no (0), did the pilot have prior glass 
experience?, and does not reflect the amount of glass flying time. 

The influence of past glass experience was tested by forming a contingency table for 
each probe (as in the SEAT variable above), and performing a chi-square test on each. 
This resulted in three rejections of the null hypothesis of row/column independence. 
The contingency tables for the three are displayed below. 
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A7. In the aircraft that I am presently flying, it is easy for the captain 
to monitor and supervise the first officer. 
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Chi-square = 8.12, df = 3, p < .05 


A1 6. I am concerned about the reliability of some of the automation equipment. 
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Chi-square = 21.05, df = 4, p < .001 


A17. I am concerned about the lack of time to look outside the cockpit for 
other aircraft. 



Chi-square = 10.47, df = 4, p < .05 
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The interpretation of the entries in the matrix is up to the reader. It would appear that in 
probe A7 the glass-experienced pilots had a narrow range of opinion, mostly agreeing 
with the sense of the probe, and a small number neutral. Those without glass 
experience showed more variability, though the distributions were centered at about the 
same place. There was very high agreement by the non-glass group - only four out of 
1 09 disagreed with the probe. 

In A16 the non-glass pilots showed a fairly symmetrical distribution, while the glass- 
experienced pilots had rather strong disagreement with the probe, with 29 on the 
disagree side and 5 on the agree side. Perhaps their experience with glass cockpit 
had relieved some of the apprehension of those making their first transition to glass. 

In A17 the glass group was symmetrically divided over the range, with most responding 
agree or disagree, and few extreme or neutral. The non-glass showed somewhat the 
same pattern, but more neutral choices. It would appear that the non-glass pilots were 
somewhat more concerned about heads up time than the glass pilots. 

Home Computer Usage 

Since the introduction of the FMS into airline fleets there has been a persistent belief 
that pilots who own a home computer profit from this experience. It was further 
assumed that it is first officers who have this exposure, giving the “computer literate” 
first officer, if not an advantage, at least some compensation for the captain’s greater 
aviation experience. We again state that this entire line of reasoning has been based 
on assumptions and beliefs, not on empirical data. 

In an earlier NASA report (Wiener, et al., 1991, p. 25), the question of ownership of 
home computers was raised. Of the captains, 71 % responded yes, and for the first 
officers it was 50%. For this sample size (n = 73), the difference was not statistically 
significant. 

In this study we corrected the mistake we had made in earlier studies by asking not 
about ownership, but usage: “Do you use a personal computer at home?” (PCUSE). 
The second question (PCFREQ) dealt with how often it was used. The statistical test 
involved a 2-by-4 contingency table (see Table IV-1a). Once again we found no 
difference between the responses of the captains and first officers. Thus we are 
convinced that the myth of the computer- literate first officer and the computer-naive 
captain is unsupported. Whatever problems captains may have in transition to glass, 
compared to the first officers, is probably not due to differences in home computer 
experience. 

There were two significant correlations to report. PCFREQ correlated significantly with 
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A1 (“Flying today is more challenging than ever”) (r = -.18). Since the correlation 
coefficient is negative, it indicates that frequent PC users (low index numbers) tended 
to have low approval (high Likert scale response values) of this probe. We find it 
difficult to interpret this result. 

Likewise, PCUSE correlated significantly with A16 (“I am concerned about the reliability 
of some of the automation equipment .”) (r = -.19). Here the result may be more clear. 
The correlation coefficient is negative, indicating that PC users (“1") tended to give 
higher Likert responses (disapproval of the probe). PC users may indeed be more 
accepting of automation technology, even its faults, than non-users (“2"). 

Summary 


These data, and the descriptive data presented in Chapter IV have not produced any 
startling results, but together paint a mosaic of the pilots’ attitudes toward transition 
training in a new technology aircraft. Further details will be found in crew members’ 
responses to the open-ended questions, presented in Chapter VI. This completes the 
discussion of Phase 1 by itself. We now turn to Phase 2, and to comparisons between 
Phase 1 and Phase 2. 
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C. PHASE 2 


The second phase of the experimental design was timed to be about four months after 
ground school, following all training including IOE, and assignment to a base to fly the 
757 line. 

Much of the data are summarized graphically in Chapter IV. The second phase 
questionnaire, Q2, included a small number of demographic variables, four open-ended 
questions (which are analyzed in Chapter VI), and a 24-item attitude scale. The 24 
items included the 20 utilized in PI , plus four new items (21-24). The following 
demographic variables were included: 


TIME757 

Total hours 757 time 

CBASE 

Current base 

ACASSIGN 

The plane the pilot that was assigned to after training 
(757 or otherwise) 

MORTRN 

The number of months assigned to plane other than 757, 
following transition training, prior to assignment to 757 line. 

PROB757 

This variable sought to measure problems encountered by 
pilots returning to their old aircraft to await a 757 assignment. 
Since so few fell into this category (see Figure IV-14), we have 
not used this variable. 


Intercorrelations 


The intercorrelation matrix of the 24 Likert scale variables, as well as the demographic 
variable TIME757 was computed. The intercorrelation matrix is too large to include in 
this report; it is available on request. 

ACASSIGN and MORTRN 

These variables are discussed and graphics depicting the variables are displayed in 
this chapter. The graphics of ACASSIGN (Figure IV-14) and MORTRN (Figure IV-15) 
indicate that only about 12 per cent of the sample was unable to move directly into 757 
line assignments. In the first few classes the figure was somewhat greater, as 


V-10 



deliveries did not keep up with pilot training. Later the opposite was true: deliveries ran 
ahead of pilot training, training was accelerated, and new 757 pilots went to the line 
without delay. 

TIME757 

The variable TIME757, the number of hours of 757 time accumulated up to the 
completion of the P2 questionnaire, is displayed graphically in Chapter IV as Figure IV- 
16. It is displayed again in the next sub-chapter of this chapter along with the same 
question for P3, so that the growth in flying time accumulated in P2 and P3, over a 12- 
14 month period can be compared (Figures IV-16 and IV-17). 

It would be interesting if this variable correlated with various attitude scale scores. The 
correlation between TIME757 and each of the 24 Likert scale variables was computed, 
and only one was significant: the correlation with A18 (“Continental’s CRM program has 
been helpful to me”) was 0.22 (p < .025). (Under the null hypothesis of zero correlation 
between two variables, for n = 102, an absolute value of Pearson’s r greater than .196 



FIGURE V-2. Attitude toward company's CRM program. 


significant at the .05 level, two-tailed test). Since the correlation is positive, this 
indicates that pilots with higher flying time in the 757 tend somewhat more to reject the 
probe. This is consistent with the finding from PI that there was likewise a significant 
correlation between total flying time (TFT ALL) and the A18 (CRM) probe (see page V- 
6). The history, background, and theoretical foundations of Continental’s CRM 
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program is discussed in Chapter VII. 

PROB757 

On Q2 there were questions about what plane the pilot returned to after 757 transition, 
if he could not be assigned to the 757. As we indicated previously, only about 16% of 
the sample returned to their previous plane rather than the 757, and this sub-sample 
was too small to be worthy of statistical testing. For confirmation, see Figure IV-14. 


GLASSNU 

24 contingency tables were formed, using the derived index GLASSNU and each 
attitude probe. These resulted in mostly 2-by-5 tables, in a few cases 2-by-4. Each 
was tested using the chi-square test. None was significant. We can conclude that 
pilots who had formerly flown glass cockpits did not differ in attitude toward training and 
automation from those who had not. 


SEAT 

Contingency tables 2-by-5 (or 2-by-4) were formed to test the variable SEAT (captain 
vs. first officer) against the 24 Likert attitude probes. None was significant. We again 
see that the attitude of captains and first officers did not differ in this sample. 

Summary 

The attitude and demographic data from Phase 2 have been analyzed in the foregoing 
sub-chapter. We now turn to comparisons of the attitude data between PI and P2. 
This is the longitudinal analysis. Out of this analysis will come a comparison of 
responses in the two phases, which will tell us whether attitudes shifted between 
examination during Phase 1 (sign-up) and second Phase 2 (post-lOE). 


D. COMPARISON OF PHASE 1 AND PHASE 2 ATTITUDE RESULTS 


In this sub-chapter we examine the attitude results from PI and P2, to determine 
whether there has been an attitude shift during the 3-4 month period between the times 
when the pilots filled out Q1 and Q2. We are particularly looking for shifts in attitudes 
toward training and toward automation in general. A shift would be indicated by finding 
differences in a pilot's responses to the same question asked during the two periods, 
that is, an inconsistency between response on P2 and P3. 
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PI vs. P2 Comparisons: Corresponding Questions 


The following adjustment was made in numbering of Q1 and Q2 probes. Q1 No. 8 (“/ 
am very apprehensive about going through this transition”) was inappropriate for Q2 
and Q3, so no comparison with it was possible. Q1 No. 14 was moved to take its 
place. On page A-8 the probes 14a, 8b, and 8c are shown graphically. Although the 
numbers are different, the probes are the same. Otherwise each of the first 20 pages 
of Appendix A show the three graphs representing the three phases in proper order 
(Probes 1-13, and 15-20). Pages A-21 through A-24 display, two to a page, the results 
of the four probes used on Q2 and Q3, but not Q1 

The Test Statistic: Marginal Homogeneity 

Since the test statistic may not be familiar to all of the readers, we shall describe it 
briefly. The statistical measure is called the marginal homogeneity test. It is an 
extension and generalization of the familiar McNemar repeated measures test with two 
response categories (2-by-2). The McNemar problem is generalized to K-by-K matrix 
for K response categories. There is also a K-by-K categories test attributed to Bowker, 
used in a previous field study (Wiener, 1 989). For the mathematical development of 
the marginal homogeneity test, see Agresti (1990). 

The data must be categorical and ordered. Arbitrarily the first phase (PI ) responses 
are assigned to rows, the second phase (P2) to columns. Thus for the attitude data, a 
5-by-5 matrix (or in some cases smaller) is produced, with cell ij representing a 
response of / to the first application of the probe (PI), and j to the second (P2). If the 
pilot responds the same on both applications of the probe, the tally will go in the main 
diagonal (I = J). If there is a shift in opinion, more cases will fall off the main diagonal. 

Page C-1 is repeated on the following page. The probe is No. 1 , “Flying today is more 
challenging than ever." Looking at the main diagonal, 15 pilots chose the “strongly 
agree” response category on both Q1 and Q2, 21 chose “agree”, etc. The off-diagonal 
tallies indicate shifts in attitude between the first to the second polling. Using the same 
example, nine pilots changed their response from “strongly agree” to “agree”. If there 
were no changes, the entire tally would be contained in the main diagonal. The 
greater the change in attitude, the further the tally would fall from the main diagonal. In 
the example, four pilots changed their attitude response from “strongly agree” to 
“disagree.” These were large defections from the initial (P2) position, but no full-scale 
changes (from “strongly agree” to “strongly agree”, or vice-versa. 


V-13 


Phase 


1 . Flying today is more challenging than ever. 



Phase 2 



Strongly 
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Results: PI vs. P2 


Under the two-tail null hypothesis of no change in attitude, the responses should be 
clustered on or near the main diagonal. A large number of off-diagonal entries (in 
either direction under a two-tail null hypothesis) would lead to a rejection. For a one-tail 
hypothesis, the direction of deviation from the first phase to the second is specified. We 
used the test on the 20 probes in common to PI and P2, and the 24 probes in P2 and 
P3. In Appendix C we have provided the matrix for only those probes that were 
statistically significant. Along with the response matrix we have provided a graphic 
displaying the frequency of response for each of the five response categories. These 
two figures, although they use the same data, do not display the same information. 

The bar graph shows trends of groups, not the choices of individual pilots. 

Table V-3 lists the eight significant marginal homogeneity tests from Phase 1 compared 
to Phase 2, and indicates the nature of the change. The full text of the probes can be 
found in Appendix D. The movement of response from Phase 1 to Phase 2 in the 
attitude questionnaires can be seen graphically in Appendix C. 

There is no consistent pattern in the movement toward disagreement with the probes, 
charted from PI to P2. Some of the probes are positively stated toward automation 
(e.g. No. 3), and some are negative (e.g. No. 2). As the pilots repeat the questionnaire 
in P2, approximately three to four months after the first set of responses in PI , a 
movement toward less agreement with the probes does not portray a consistent attitude 
toward automation. The pilot changes his choice toward less agreement with No. 11, 
looking forward to more automation, and likewise changes toward less agreement with 
the more negative No. 1 3. 

Perhaps we will find the next set of marginal homogeneity tests, for P2 vs. P3, more 
instructive. In those tests the pilots will have had some flying experience in the 757, 
limited to a few months in P2, and over a year’s worth in P3. In any event, it is 
interesting to note the volatility of opinion in PI vs. P2, that eight out of 20 probes 
should result in statistically significant changes of opinion, even if we cannot find a 
consistency or an underlying meaning in these opinion shifts. When we compare 
opinions in P 2 and P 3, we would expect more stability of opinion, that is, fewer 
significant changes. 
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TABLE V-3. Statistically significant marginal homogeneity tests. 


Probe Abbreviated Text 

1 Flying is more challenging than ever 

2 Concerned that automation will cause 
skill loss 

3 Automation leads to safer operations 
7 Easy for captain to monitor first officer 
1 1 Look forward to more automation 

13 Too much workload terminal area 
15 Easy to bust altitude today’s environment 
17 Concerned about time to look outside 


Movement: PI to P2 
Toward disagreement 
Toward disagreement 
Toward disagreement 
Toward disagreement 
Toward disagreement 
Toward disagreement 
Toward disagreement 
Toward disagreement 
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E. PHASE 3 


Phase 3, the final phase of the longitudinal study, was designed to collect data from the 
remaining volunteers at a time when they had about one year of line experience, or 
about 16-18 months after initially joining the study on the first day of ground school for 
757 transition. We felt, based on past experience in field studies, that at this time 
opinions would have solidified, and would probably not change appreciably if the 
interval between P2 and P3 were extended. Also, we would expect that at this time the 
pilots would feel “comfortable”, a word widely used by pilots to describe their feelings at 
some experience level after transitioning to a new aircraft. To be “comfortable” in the 
new aircraft would mean that the pilot was free of apprehensions about his ability to 
manage the cockpit and particularly the automated features, to be able to respond 
appropriately to non-normal situations, and in brief to feel at home, relaxed, self- 
confident, and in command of his own abilities.. 

Phase 3 was designed mainly to obtain the final data on the 24 attitude probes. There 
is a minimal amount of demographic data sought on the questionnaire (see Appendix 
D), and a minimal number of hypotheses to be tested. Most of the data are merely 
descriptive. 

Miscellaneous Questions 


As to the “comfort” dimension, 91 of the 94 valid answers were “yes” to4he question 
“Do you feel comfortable’ in the 757 now?” 

The data for the pilots’ current base are tabulated below. The number of responses to 
this question of IAH and EWR are about proportional to the 757 flying time assigned to 
those two bases. The desirability, from the pilots’ view, of Continental’s various bases 
is discussed elsewhere. The Guam 757 base was essentially ignored in this study, 
since it was formed after the study began. 


IAH (Houston) 33 

EWR (Newark) 56 

LAX (Los Angeles) 1 

Other 6 

Total 96 
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Cockpit Positions Held 


The cockpit positions held by the volunteer pilots is shown in Table V-4. Note that the 
96 pilots represented by this table were the larger group (82 per cent) who were still 
flying the 757; the remainder were flying other aircraft. As a percentage of the 149 
original volunteers, this group is probably under-represented due to the fact that many 
pilots, after bidding off the 757, dropped out of the study and did not send in Q3. 


Table V-4. Current Aircraft and Seat of 96 Pilots. 
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Contingency Tests 


To test the null hypothesis that attitudes, as measured by our Likert scales, are not 
related to SEAT (position in the cockpit), 2-by-5 (in some cases 2-by-4) contingency 
tables were cast, as in Phase 2. Chi-square tests were performed on the 24 resulting 
tables (one for each attitude probe). None was significant. We must again conclude 
that captains and first officers in this sample saw things alike. 

Similar tables were set up using the derived variable GLASSNU (previous glass 
experience, or none) as one variable, responses on the Likert scales the other. None 
was significant. We conclude that pilots with past experience on glass aircraft, and 
those with no such experience, held similar attitudes as measured by our scales. 

Intercorrelations 


An intercorrelation matrix containing responses on the 24 attitude scales was formed, 
but was not examined statistically. 


F. COMPARISON OF PHASE 2 AND PHASE 3 ATTITUDE RESULTS 


As in sub-chapter D, we shall now examine the 24 attitude scales for possible shifts in 
attitude from P2 to P3, using the marginal homogeneity tests. 

Of the 24 attitude scale items, three showed significant changes from P2 to P3, as 
summarized in Table V-5 below. 


TABLE V-5. Statistically significant marginal homogeneity tests, P2 vs. P3. 


Probe 

Z 

Text 

Movement P2 to P3 

17 

-2.3 

1 am concerned about the lack of time to 
look outside the cockpit for other aircraft 

Toward agreement 

18 

-2.7 

Continental’s CRM program has been helpful 

Toward agreement 

24 

3.0 

There are still modes and features of the 
757 that 1 do not understand 

Toward 

disagreement 
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As Table V-5 and the figures in Appendix E indicate, there was a significant shift of 
opinion in the pilots in the roughly 14 months between Phases 2 and 3. Numbers 17 
and 24 indicate movement favorable to the 757 flight guidance system, and perhaps 
toward automated flight in general. In previous field studies these are common worries 
of the new FMS aircraft pilots: sufficient time for extra-cockpit scanning; and mode 
confusion, as it has come to be called. 

We observed also an increasingly favorable view toward the company’s CRM program. 
We can only speculate as to the reason for this. It is most likely due to the emphasis 
put on CRM throughout the 757 transition training. Both fleet managers insisted that 
CRM be taught and included as part and parcel of the flight training, not as a separate 
block of instruction (see quotation, top of page VII-1 ). Some authors have stated 
previously (Wiener, 1989) that good CRM practices are even more essential in the 
automated than in the traditional cockpits. This point was emphasized from the first 
day (H-CAT training). Much of the credit for the emphasis on blending CRM with flight 
training goes to instructors at Boeing. It was there that the early cadres of Continental 
instructors first encountered this method of training. Typically flight training and CRM 
are taught as two worlds apart. Seeing the merit of the Boeing approach, it was 
transported back to Houston and made part of the flight training program. Presumably, 
over the period between filling out Q1 and Q2, a space of over a year, the pilots had an 
opportunity to practice their CRM skills in a demanding, automated environment. This 
perhaps accounts for the shift toward a more positive response to probe number 1 8. 

Usage of HSI Modes 

At the request of Continental flight management, we included a question in the third 
(final) questionnaire (Q3), asking the 757 pilots the percentage of time they used, in 
flight, the four HSI modes (map, expanded VOR, expanded ILS, and compass rose). 
See Page D-1 0. 

The map mode was used about 95% of the time, and the remaining 5% was about 
evenly split between the other three modes. As we have noted in previous studies (e.g. 
Wiener, 1989), most pilots will, if unimpeded, fly an entire trip with the HSI in map 
mode. Why not? It is a spectacular example of what can be done using glass displays. 
Only a non-precision approach would move the crew to select a different mode. 
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VI. OPEN-ENDED QUESTIONS AND RESPONSES 


A. INTRODUCTION 


The intent of this chapter is to take the reader into the “details” of Continental’s 
B-757 training program. The open-ended responses and summaries which 
follow are an attempt to capture the experiences and comments, in their own 
words, of pilots progressing through their transition training, IOE, and eventually, 
flying the line. No attempt has been made to place judgment on the responses 
with respect to appropriateness, quality or significance, but only to categorize 
them for descriptive analyses. The groupings and summaries are presented in 
the context of the training program with an emphasis on topics such as 
problematic areas, recommendations for improvement, effectiveness of the 
training aids, and the implications for pilots in transition training. In this way, the 
reader can make his or her own judgments on the responses. 


B. METHODOLOGY 


The challenge of collecting qualitative data is to reduce it into a manageable and 
meaningful format, and then make sense of it, especially when it comes in 
voluminous amounts. This study was no exception, particularly when one 
considers that a set of open-ended questions were asked of each pilot just after 
their B-757 training, and then again after approximately one year of flying the 
line. This resulted in querying over 100 pilots twice on the following four topics: 
(1) training for the B-757, (2) errors observed or committed on the line, (3) crew 
coordination and procedures, and (4) cockpit workload. In addition, there was a 
question for those who left the B-757 pertaining to their reactions on having left 
the B-757. These five topics provided the initial structure for presenting the 
responses. 

Once the data were organized and placed in a coherent structure, the task of 
identifying trends and regularities was proceeded. There are many ways to 
identify regularities in qualitative data but the method chosen for this study was 
to further characterize the responses by conducting an inductive analysis. In this 
way, the data defined itself by having the patterns and characteristics emerge 
out of the chaotic responses. This was felt to be the most appropriate approach 
rather than imposing structure upon the data, especially with respect to 
identifying training problems as well as making recommendations and proposing 
intervention strategies. 
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As previously mentioned, one of the initial difficulties encountered in this study 
was the voluminous amounts of open-ended data collected. This was further 
compounded by the somewhat arbitrary nature of some of the responses and the 
applicability to multiple categories (e.g. cockpit workload and CRM). Another 
consideration was the longitudinal nature of the study. An attempt was made to 
look at each questionnaire individually and then summarize the topic as a whole. 
We felt that this approach would capture any specific patterns after the initial 
transition training in Questionnaire 2 (Q2) and then once again after flying the 
line for approximately one year in Questionnaire 3 (Q3). In addition, Q2 and Q3 
topics were summarized together at the end of each major question section. 

Initially, the responses were transcribed from the individual questionnaires into 
an electronic format for ease of manipulation. The quotations are as close to 
verbatim as practical, with some minor editing such as punctuation, spelling and 
improvement to the flow of the wording. Several responses contain editorial 
insertions by the authors and are enclosed by these symbols < >. In addition, 
exclamation marks, question marks, and words underlined for emphasis are the 
work of the respondent and not the authors. 

Once the responses were electronically transcribed, they were grouped 
according to questionnaire number (Q2 or Q3) and placed in the appropriate 
topic (1 ) training, (2) error, (3) CRM, (4) cockpit workload, and for a select few, 

(5) initial reaction having left the B-757. After grouping, an attempt was made to 
analyze the responses with a computerized narrative analysis tool called 
Quantitative, Objective, Representative, Unambiguous Modeler (QUORUM; see 
McGreevey, 1995, 1996, 1997). For a further description of the QUORUM 
method see Appendix I. The QUORUM results on the open-ended responses 
were inconclusive, due to the short length and minimal narration in the pilots’ 
responses. 

Due to the inconclusive results of QUORUM, a manual sort was undertaken to 
develop the character of the responses. The four main topics were further 
decomposed into the individual question components as follows: 

1.0 Training 

Questionnaire 2 

1.2.1 What did you think of your training for the 757? 

1 .2.2 Did you have trouble with anything? 

1 .2.3 What topics should receive more or less emphasis? 

1 .2.4 Please comment on the training aids and devices. 



Questionnaire 3 


1.3. 1 What did you think of the training you received for the 757? 

1.3.2 Is there any way you would recommend it to be changed? 

1.3.3 Did the training program (including IOE, LOFT, etc.) prepare you to fly 
the line? 

2.0 Error 

Questionnaire 2 and 3 

2.1 Describe in detail an error which you have made, or have seen someone 
else make, with the automation, that might have led to some undesirable 
consequence. How could it have been avoided? (equipment design, training, 
CRM, procedures?) 

3.0 CRM 


Questionnaire 2 and 3 

3.1 What can you say about crew coordination and procedures in the 757? 

3.2 In what way are they different from previous planes you have flown? 

3.3 What areas can use improvement? 

4.0 Workload 


Questionnaire 2 and 3 

4.1 How would you compare the overall workload in the 757 compared to your 
previous plane? 

4.2 Please mention anything that you feel should be changed to help you 
manage workload (procedures, ATC, training, etc?). 

5.0 Departed the 757 

Questionnaire 3 (only) 

5.1 After you left the 757 and went to another aircraft, what was your reaction? 

5.2 What did you miss about the 757 avionics and automation? 

5.3 What did you like better about the older technology planes? 

5.4 Plane and seat you went to: Aircraft Seat 

Responses to each question were entered into tables according to a major 
keyword in the response (e.g. for error: “procedures”). Some responses required 
another entry according to a minor keyword in the response, if applicable (e.g. 
“procedures” AND “not following”). With this “keyword” method, it was a matter 
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of cutting and pasting the responses into the appropriate categories and then 
observing the patterns which emerged. 


C. OPEN-ENDED RESPONSES 


Introduction 


Each open-ended response topic (1) training, (2) error, (3) CRM, (4) workload, 
and (5) departed the B-757 (if applicable), was examined individually according 
to the questionnaire number (Q2 or Q3) and then again, in a combination of both 
questionnaires (Q2 and Q3). Since the authors wish not to burden the readers 
by presenting all of the comments and responses received, only those comments 
which are typical, contrary, or unusual in nature will be presented. However, all 
of the comments and responses are available for qualified researchers by 
contacting the authors. 

1.0 Training 

1 .2.1 and 1 .3.1 What did you think of your training for the 757? 

Questionnaire 2 

Of the 84 pilots who specifically stated their reaction to their training experience, 
the overwhelming majority stated that the training program was excellent. In 
addition, there was no mention of dissatisfied trainees nor any dislike of the 
program. 

• Excellent/best training I’ve had/outstanding/great. (45)* 

• Good/very good/effective. (29) 

• Adequate. (4) 

• Inconsistent. (6) 

‘Number in parenthesis indicates similar responses. If none are present, then 
the reader may assume only one response of that type. 

Questionnaire 3 

Again, of the 62 pilots who specifically responded to the question, the vast 
majority felt the training program was good or excellent. 

• Excellent/best training I’ve had/outstanding/great. (39) 

• Good/very good/effective. (20) 

• Adequate. (3) 

• Inconsistent. (0) 
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Q2 and Q3 Summary 

It is a commendable achievement for Continental’s training department to 
receive such high accolades for their B-757 training program. There was not a 
single pilot who stated that he was dissatisfied or felt the program was inferior, 
which is in contrast to prior reports on automation and training programs (see 
Wiener 1989, BASI 1998). Even after approximately a year on the line, the pilots 
were exceedingly satisfied with their training program. 

1 .2.2 Did you have trouble with anything? 

Questionnaire 2 

There were few direct responses to this question. Most pilots addressed what 
should be improved or which topics needed more emphasis. The following three 
characteristics emerged from those who responded: 

Felt rushed, intimidated, or uncomfortable (8) 

• I felt rushed. (5) 

• I felt uncomfortable the whole ground school. 

• There was so much material in so short of time that I am reviewing my 
manuals to re-leam all that I missed. Note: re-leaming is different than 
reviewing. 

• Difficult and intimidating. 

Autoflight mode confusion (3) 

• I had trouble adjusting to the use of different auto flight modes and 
some confusion as to which button to push and which mode to use for 
different aspects of flight. 

• VNAV path is an area that rarely operates as I think it should, probably 
because I do not fully understand what it is using to make its 
decisions. 

• I didn’t understand a few things initially with LNAV and VNAV, but 
didn’t quite know just what I should know or ask. 

Oral exam (2) 

• I had problems preparing for my oral exam. (2) 

No problems or troubles (so stated). (9) 
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Questionnaire 3 


Once again, there were few direct responses to this question. Of those who 
responded, the topics where as follows: 

Felt rushed or intimidated (6) 

• The company tried to squeeze a lot of new aircraft, new technology, 
and procedures in to too short of a time span for a rating ride. 

• Fast paced and intense. 

• Very difficult and frustrating. Too much, too soon, and too fast. 

• Too rushed. 

• Felt intimidated by the automation. 

• I have not received “training” only what is required to fill the FAA 
requirements for training. 

LOFT and IOE (2) 

• Most problems during IOE and LOFT seemed to occur with the pilots 
who did not read the manual. 

• My only LOFT was an Atlantic crossing, which I have never done in an 
aircraft. I would have benefited from a domestic LOFT. 

Q2 and Q3 summary 

As previously stated, few pilots responded directly to this question. Most of 
those who replied to this question had suggestions or ideas to improve the 
program (see next section). Of those who did, the feeling was that the training 
program was rushed and contained too much information for such a short period 
of time. This seems especially true of those with no previous glass experience. 

1 .2.3 What topics should receive more or less emphasis or should be 
changed? 

Questionnaire 2 

There were numerous pilot responses to this question. 

Instructors (4) 

• Outside instructors <non-Continental personnel need to be pilots or 
trained on our standard procedures. 

• There needs to be some scheduled time with an instructor every day 
just on systems. 

• Instructors lacked confidence. 

• Instructors applied pressure to learn procedures. 

“On-the-line” learning (5) 

• Too much emphasis on OJT <on the job trainings (2) 
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• The expectation of the training department that minimal exposure 
received in training should be adequate is wrong. 

• I noticed that other students with no FMC background having a harder 
time with line operations. 

• It <training> generally came together on the line with lots of practice 
using the equipment. 

Instructional topics (16) 

• I am still not up to speed on programming the FMC. (2) 

• I would like more in depth systems knowledge. (2) 

• Windshear training verged on overkill. 

• Training on the CDU was almost non-existent. 

• More emphasis should be placed on the aircraft flight manuals. 

• The FBS was over utilized in my case (5 years on the B-737-300). 

• A more in depth explanation of the IRS’s function could have been a 
help. 

• The training was lacking nuts and bolts. 

• Instruction in Long Range Navigation was too deep, the experienced 
pilots knew better and the domestic pilots were “in shock” - teach the 
basics and keep it simple! 

• There needs to be a greater emphasis on CRM and the greater need 
for the crew to interact with the automation (FMC) and each other to 
preclude mistakes. 

• Human-automation interface training would have been more 
meaningful to me if it had been given after the sim training instead of 
before ground school. 

• Exposure to automation should be done before training for those 
without prior experience. 

• The FMC training needs to be focused on “real” operational situations. 

• Being computer literate made the FMC a breeze to understand. 

Questionnaire 3 

Instructors (5) 

• Need a higher level of experience on the part of the simulator 
instructors. 

• Good instructors and check airmen. They are out to help rather than 
“grade” the pilot. 

• Everyone involved was visible, available, and helpful, but certain once 
the program is fully integrated into our IAH facility, that will 
unfortunately change. 

• The captains, simulators, and instructor teaching was excellent. 
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“On-the-line” Learning (5) 

• Training gave us the push-button knowledge, but flying the line was 
the teacher. (3) 

• For the most part the training prepared me for line flying, but actually 
being on the line in everyday operations and utilizing all of the 
information brings it all together. 

• More CDU training and operation as it took 3 months on the line to feel 
comfortable with the various modes and automation. 

Instructional topics (6) 

• More time with the automation. 

• Specify strict procedures for only one pilot to program the FMC/MCP 
below 10,000. 

• More preparation for the oral. 

• Need more time spent on VNAV operation and profiles. 

• Too much emphasis on FMC programming. 

• IOE should be after 1 00 hours <on the line> especially if this is your 
first glass cockpit. 

Q2 and Q3 summary 

This section is where one starts to see some divergence in pilots’ responses with 
respect to the FMS and the automation in general. The FMS training seems to 
be either insufficient or excessive in some cases. Whether this is related to any 
prior glass cockpit experience or not is unknown. However, some respondents 
state that they have had prior automation experiences with aircraft such as the 
B-737-300/400. Of some concern are the responses which refer to “on-the-line” 
or On-the-Job-Training (OJT) and “outside” or non-Continental (non-CAL) 
instructors who were not practiced in company SOPs. These two situations, OJT 
and non-cal instructors, may leave a pilot with some ambiguity in certain 
situations and, as a result, pilots may reinforce erroneous actions or faulty 
assumptions. 

1 . 2.4 Please comment on the training aids and devices 
Questionnaire 2 

Positive comments (39) 

• CBT is excellent/very effective. (13) 

• The CBT allowed students to progress at their own pace, and review 
material. (9) 

• Training aids were good/very good. (5) 

• The CBT along with the FBS was very exciting/impressive. (4) 

• Training aids and devices were adequate. (4) 

• CBT and training devices in a building block approach is quite effective 
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• The CBT was the best I ever received. 

• The training aids and devices were state of the art. 

• The CBT was better for not having to listen to an instructor ramble or 
go off on a tangent. 

Negative Comments (58) 

• We need a FMC/CDU training aid for practice. (28) 

• The CBT aids were inconsistent and/or had errors. (17) 

• One needs to be able to go directly to a specific item instead of 
listening to a large portion of a system to answer one question. (3) 

• I did not like the CBT, very impersonal and boring. (3) 

• There needs to be more questions and answers on the CBT. 

• The CBT is not the best way to leam an airplane. 

• CBT was slow and frustrating to use. 

• The CBT training is linear oriented and does not encompass the 
complete scope of the automated systems. 

• At times, I had to “figure out” what the computer answer was rather 
than the system comprehension understanding in order to progress, 
which is negative learning. 

• The CBT lulled one into a false sense of confidence. 

• The FBS should not be used as a substitute for a real sim. 

Questionnaire 3 

Positive Comments (5) 

• The CBT was excellent. (3) 

• CBT, FBS, and then the full sim was a logical progression. 

• The CBT was the best of my career. 

Negative Comments (19) 

• There needs to be an operating CDU/FMC trainer. (13) 

• The CBT was distracting in several areas because it had errors. (2) 

• The CBT could be improved to allow for more realistic FMC 
operations. 

• CBT (in lieu of instructor-led ground school) is sheer drudgery. I’m 
sure that my retention of systems and overall understanding of 
systems operations is significantly lower on the B-757 due to CBT. 

• Very dull in the personal computer trainer. 

• The fixed-base simulator looked like an expensive make-work 
government project. It is not a simulator, but treated as a simulator, 
and graded as one, to the detriment of the student. 
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Q2 and Q3 summary 

Many of the pilots responded that the CBT was a good or excellent training 
device. However, some pilots mentioned the CBT contained errors or that they 
had experienced frustration by having to retrace their steps in order to review a 
particular topic. In addition, many pilots mentioned a desire for a workable FMS 
trainer which would enable them to practice building FMC programming skills 
and techniques outside of the simulators (fixed base and full motion). 

These last three issues: (1 ) CBT errors, (2) wading through prior CBT material in 
order to review previous topics, and (3) the need for an operational FMS trainer, 
raise some concerns with the authors. Errors in the CBT are inappropriate for 
pilots undergoing transition training or any other type of airline training for that 
matter. In addition, one can empathize with a pilot’s frustration by having to 
navigate through prior material in order to review a previous topic only one or two 
frames away. Finally, we suspect the requests for the FMS trainer may be 
predominantly from pilots without prior glass experience. 

1.3.3 Did the training program (IOE, LOFT, etc.) prepare you to fly the line? 
Questionnaire 3 (only) 

Twenty eight pilots (28) responded that the training they received adequately 
prepared them for flying on the line. There were no negative responses to this 
question and many without a response indicated. 

General comments on B-757 training program 

Questionnaire 2 

Style of Instruction 

• Felt the training was “bought cheap” and not kept up to date with 
changes or new information. 

Curriculum Development and implementation 

• Coming off the B-737-500/300 made the training easy. (3) 

• Coming from the B-737-300/500, it seemed more like transition 
training. 

• I came from the B-737-300/500 and was bored with some simulator 
sessions - I should have had a "short course.” 

• Previous experience on the B-737-300/500 made the transition 
extremely easy. My only negative comment would be the length of 
training seemed a little long. 

• Coming from the B-737-300/500, 1 had the advantage of being familiar 
with the glass cockpit and FMS computer which helped me a lot and 
made the B-757 training much easier. 
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• I was a Captain on the B-737 and went to FO position on the B-757. 
The change of seats was more trouble than the aircraft change. 

• For pilots that have never had an FMS aircraft prior to the B-757, it 
requires a lot of hands on training. 

• The transition from the B-727 was a quantum leap. 

• I noticed students with no prior FMC background had a more difficult 
time with training. 

Administration and Scheduling 

• The 14 hour day needs to be reduced. 4 to 6 hours training/day with 
study time would equal an 8 -10 hour day. 

• I wish I could have obtained my manuals sooner for studying. 

• Being paired with the same FO throughout the training sessions was 
helpful. We lived together, studied together and flew together. Big 
benefit. 

• They did not give us enough (almost none) information about training 
before-hand. 

• ETOPS training should be given after IOE. 

Questionnaire 3 

Style of instruction 

• Would like to see a group class. 

• Bring back the classroom environment to create the question/answer 
exchanges from other pilots. 

• If the line environment was as exciting as the training, I would have 
stayed on the aircraft. 

• The LOFT training was invaluable. It all came together in those 
sessions. 

• We need a LOFT program for training (had only one after PC) and 
more recurrent training (we have none). 

• The B-757 training was a self-taught course with too much verbiage in 
the supplemental training guide. 

Curriculum development and implementation 

• Solicit feedback from the pilots. 

• I came off the B-737-300 which is also automated. I thought the 
transition to the B-757 was easy. 

• No complaints except a shorter course should be offered for pilots who 
transition from B-737-300/500. 

• Training was well standardized and positive in nature. 

• The simulator and LOFT sessions were very good. 
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Administration and scheduling 

• The <oral> exam would have been more relaxed somewhat if some of 
the FMC work was saved until after the oral. 

• Providing the study manual and flight manual before beginning training 
to give the pilots a chance to prepare ahead of time. 

• I feel that training someone on any equipment, then letting them sit for 
three months is extremely dangerous and stupid. I lost currency twice 
before I logged 100 hours. Floundering around in an unfamiliar 
cockpit, trying to take in the finer points of long range navigation and 
skirting 23,000’ mountains on the backside of the clock is not my idea 
of a good time. 

Training Summary 

Once again, Continental’s training department receives high accolades for such 
a positive response to their B-757 training program. Certainly some areas could 
use improvement, but the majority of the pilots felt their training gave them the 
skills and information necessary to fly the line. However, it is in this training 
section that one starts to see a dichotomy between those pilots with no prior 
glass experience and those with previous glass experience. This dichotomy is 
especially prevalent in section 1.3.3 General comments on B-757 training 
program. In these general comments, one finds pilots with prior glass 
experience (mostly B-737-300/500 aircraft) commenting that the training was 
easy or in one instance, “boring.” On the other hand, one can sense some pilots 
struggling with learning a whole new concept of flying and learning the FMS 
associated with glass cockpits. 


2.0 Error 

Introduction 

The responses to this question were read, sorted, and then categorized 
according to the type of error. While many responses indicated that the error 
had been committed by the respondent, some responses were instances were 
the pilot observed an error either from the cockpit or jump seat. Several of the 
responses were complaints or irrelevant comments and these were discarded 
from the categorization. In addition, an error was placed in only one category 
with no multiple entries. 

2.1 Describe in detail an error which you have made, or have seen 
someone else make, with the automation that might have led to some 
undesirable consequence. 

This topic was handled differently from the other open-ended responses, in that 
all the responses from questionnaire 2 and 3 were merged to derive the error 
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topics. Once the errors were sorted and categorized, the responses were placed 
back into their respective questionnaires (Q2 or Q3). A total of 12 error types 
emerged from the response sort and analysis with the following topics emerging: 


Error TvDe 

Total 

= (Q2 + Q3) 

2.1.1 Programming CDU/MCP 

(incorrect, incomplete, neglecting or not cross-checking) 

52 

(29 + 23) 

2.1.2 Energy management 

(altitude bust, not meeting speed or crossing restriction) 

24 

(16+ 8) 

2.1.3 Automation 

(over-reliance, surprise, or loss of situational awareness) 

23 

(8 + 15) 

2.1.4 Action 

(out of sequence, neglected or incorrect) 

19 

(6 + 13) 

2.1.5 LNAV 
(setup/confusion) 

14 

(6 + 8) 

2.1.6 Mode switching 

(confusion with mode switching or current mode state) 

14 

(9 + 5) 

2.1.7 Procedures 

(incorrect, incomplete, or neglecting) 

14 

00 

+ 

o> 

2.1.8 Workload 

(time for scan, distractions, or excessive heads-down) 

13 

(5 + 8) 

2.1.9 VNAV 
(setup/confusion) 

12 

+ 

00 

2.1.10 Approach 
(setup/confusion) 

10 

(7 + 3) 

2.1.11 Equipment 

(aircraft systems configuration or NAV displays) 

8 

CM 

+ 

CD 

2.1.12 Training 
(negative transfer) 

6 

(4 + 2) 


The following pilot responses are typical of the errors or incidents which were 
either committed or observed. All the error responses are not included so as not 
to burden the reader with repetition. Any suggestions of how the error could 
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have been prevented (e.g., via equipment design, training, CRM or procedures) 
took precedence and appear in the transcribed responses below. 

2 . 1.1 Programming CDU/MCP (52) 

Questionnaire 2 

• Wrong fixes entered into the computer. However, <errors> are easier 
to see in the glass cockpit. 

• A mistake that is being made by all in programming the route. If you 
are cleared EWR to LAX on Route 006, and you try to install Route 
006, but a message appears "Route does not exist.” Instead of going 
to the “Route Page” and manually placing the route in, everyone is 
trying Route 001, Route 002, Route 003, etc. until they found a route 
that matches 006. 

• The speed knob is often mistaken for the heading knob and vice 
versa. On take-off and climb-out this can cause a decrease in 
airspeed at a critical time, or the start of a turn when it is not desired. 
This is an equipment design problem; they (knobs) are too close and 
too similar in appearance. 

• We loaded the FMC manually, then the CDU kept flashing that it 
wanted to be loaded automatically by uplink causing difficulty for the 
flight. We were concerned we had loaded it manually and didn’t know 
what the consequences would be. 

Questionnaire 3 

• Most common error for all pilots is not checking the FMA after 
selections or <after> engaging a mode selected on the MCP. This 
error can be avoided by making sure that what you have selected is 
enunciated correctly on the FMA. 

• Captain entered the holding course incorrectly! He used the radial 
verses the inbound course. He was a little weak on the FMC. 

• Setting the wrong altitude in the window <MCP panel>. 

2 . 1.2 Energy management (24) 

Questionnaire 2 

• Depending on the aircraft to ensure meeting restrictions have twice 
resulted in potential violations. 

• In LVL change with 315 knots below 10,000. 

• I have never been involved in an "altitude bust” in 23 years of aviation. 
I was involved in one excursion and one trip later, almost another 
excursion for the same reason. The captain was flying both times. 
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He decided to hand fly in VFR conditions out of a high density airport. He 
failed to brief his actions and responsibilities. We were advised several 
times on climb-out of VFR traffic. After performing my duties inside the 
aircraft, I turned to the outside to look for traffic and failed to cross-check 
the captain who I had never flown with before, but was an instructor and 
at least a check airman. I assumed he was flying the airplane. 
Unfortunately, he was “outside” the airplane and had such a high rate of 
ascent that he “busted" the altitude level-off by more than 800’. My very 
next trip was with a different captain but almost the same scenario except 
I called 500' before level-off and again with a high rate of ascent. The 
captain thanked me for the notification. There is a very big need for more 
communication in these high workload areas. I have learned from these 
mistakes. I only hope the captains, who set the CRM pace in the cockpit, 
also learn from their mistakes. 

Questionnaire 3 

• Relying on VNAV path to accomplish required altitudes at certain 
waypoints. VNAV is improperly programmed for the B-757 engines 
that CAL uses. 

• Altitude busts. This A/C is so geared to smoothness for level off that if 
intermediate altitude is quickly selected, it is time to disconnect. 

• LAX CIVET arrival. I set hard altitudes for numerous step-downs 
<while> operating on LNAV. Busted 1 ,000’ below altitude at one VOR. 
First time in 22 years of flying that I had to file a NASA report. 

2 . 1.3 Automation (23) 

Questionnaire 2 

• We were cleared for a visual approach to a parallel runway while on a 
base leg (FO flying). The FO should have disconnected the 
automation and turned into the runway. Instead, he stayed on the 
base leg course and intercepted the ILS at an 80 degree angle. We 
flew near the approach course for the parallel runway. The FO was 
too dependent on the automation. 

• When I programmed in an approach, the path depicted for the IAF turn 
was not what I expected or could believe, nor did I feel the A/C would 
comply with the descent profile. 

Questionnaire 3 

• I flew one flight with the autothrottles inoperative. While encountering 
a mountain wave, airspeed blew off to a value near top bug. We were 
at FL 410 and airplane could have stalled. The autopilot kicked off for 
some reason, which is what got our attention. Extra engine power of 
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B-757 allowed A/C to recover without having to trade altitude for 
speed. Dependence on the autothrottle system took away a set of 
flight parameters that I would normally monitor without this kind of 
system and which I seldom pay attention <to anymore>. My throttle 
techniques are rusty. 

• After arriving at the LAX terminal area, ATC, due to excessive traffic 
and their inability to deal with increased traffic, asked us to turn and 
intercept the final course for rwy 24R in the north complex. This would 
have simply consisted of dialing in the corresponding ILS freq and land 
on the corresponding and assigned rwy. The FO felt compelled to 
reprogram the FMC for the ILS approach to that <new> rwy. All crew 
interactions were suspended until he accomplished “the task” of 
reprogramming the computer. I repeatedly asked <him> to dial in the 
ILS freq to the reassigned rwy. He became “hypnotized” and would 
not acknowledge my requests. His concentration on “pushing the right 
buttons” caused a breakdown of crew communications, loss of 
situational awareness and left me to fly the A/C, talk on the radio, set 
the flaps, etc. I have flown with this individual on other A/C and his 
behavior, I feel, is unique to this A/C. He seemed surprised that after 
he came back we were so close to landing. Training must emphasize 
a threshold of priority in order to fully maintain an integral crew during 
last minute changes. 

2 . 1.4 Action (19) 

Questionnaire 2 

• The one I have to work on is select LNAV after having been on an 
assigned heading after cleared direct to a fix. I’ve missed that on a 
couple of times. 

Questionnaire 3 

• Multiple instances of failing to engage LNAV after programming a 
direct track and slow to recognize that the AC is in HDG mode. 

• I was given a direct to a fix after being given a vector. After inputting 
the fix to which I was given the “direct to", I failed to select LNAV. 

After about 5 minutes, I noticed the A/C on the map display was 
deviating from the displayed route. 

• Captain forgot to activate and execute a route in the FMS. I did not 
notice until ready for take-off. Could have been avoided if one of us 
had cross-checked the other’s work. 
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2 . 1.5 LNAV (14) 

Questionnaire 2 

• Twice, since I have been in the B-757, 1 have been surprised to see 
which lateral mode is <displayed> in the flight mode annunciator. 

Questionnaire 3 

• Entering a waypoint way too close to or just past the fix and having the 
aircraft start a 360 degree turn. <l> switched to heading select. 

2 . 1.6 Mode Switching (14) 

Questionnaire 2 

• VNAV switched to FLCH. I did not realize it had done so and I was not 
watching the step downs. I was contacting the company and expected 
the automation not to miss a beat. 

• Even with my experience, it is very easy to forget to cross-check that 
the aircraft is in LNAV and not heading select or VNAV speed and not 
path. VNAV is more critical than LNAV. An aural warning when VNAV 
changes to SPD from path would be nice. Constant cross-checking is 
imperative. 

Questionnaire 3 

• I have seen multiple occasions when the VNAV system defaults from 
VNAV path to VNAV speed. This is so subtle that it is many times 
unobserved. On previous FMC aircraft that I have flown (B-737-300), 
the FMC gave a message “unable path descent”, which gave the pilot 
much better info compared to the subtle FMA change in the B-757. 

2 . 1.7 Procedures (14) 

Questionnaire 2 

• The opposite pilot executing the CDU without the knowledge of the 
other pilot. 

• Checklist items missed. 

Questionnaire 3 

• Not briefing a contingency approach for airport with low visibility (below 
1 ,200 RVR). <Suggestion:> brief multiple approaches and do not 
allow approach control to give unrealistic commands. 
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2.1.8 Workload (13) 
Questionnaire 2 


• During TO and LND phases, as well as VFR, few pilots look outside. 
Specifically when ATC assigns “Turn left to...” the first action should 
be to glance to the left in the direction of the intended turn. Most pilots 
reach for the MCP or CDU first. 

Questionnaire 3 

• <We> had a runway incursion at SFO. <l> had a new FO and <he 
was> new to the automated A/C. A chime with ACARS sounded at the 
same time <a> clearance to cross 28L and hold short of 28R <was 
received>. As captain, I heard to position and hold 28R. FO read 
back hold short and became engaged in inserting a delay code in 
ACRS (which was not working) while I taxied into position. Tower said 
nothing. <Another> A/C was on 5 mile final and had to go around 
(NASA report filed). 

2.1.9 VNAV (12) 

Questionnaire 2 

• In the VNAV descent mode the automation does not control the 
airspeed very well. 

• I saw a pilot try to use the cruise page to initiate a descent in VNAV 
and get confused because it did not present the expected information. 

Questionnaire 3 

No responses. 

2.1.10 Approach (10) 

Questionnaire 2 

• ILS capture problems in Mexico City and Bogota Colombia. 

• After selecting approach mode to ILS 25L, LAX approach controller 
changed approach to ILS 24R with a heading intercept. We had not 
practiced an approach change after all three autopilots were engaged. 
We went HDG SEL with no reaction from the aircraft. It took both of 
us about 5 seconds to finally disconnect the autopilots and hand fly the 
AC to the other runway and reconfigure for the approach. Also, when 
we were told to change runways, we were told to maintain 4,000’ until 
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intercept, but GS had been captured. If we had not disconnected the 
AP, we would have busted the 4,000’ restriction which is not good! So, 
I suggest more emphasis on runway changes after APP mode is 
selected. 

Questionnaire 3 

No responses. 

2.1.11 Equipment (8) 

Questionnaire 2 

• Problems with several different pilots having confusion with heading up 
verses track up (map mode). 

• I accidentally turned on the APU in heavy rain on short final because 
the two switches (APU and wiper) are identical and too close together. 

• I arrived at the runway with the flaps up (for take off). The 
<unextended> flaps <were> discovered in the take-off checklist. 

• Almost missed a change in flap settings. We use 20 degrees 99% of 
the time, and grow accustomed to that. When it’s changed it’s easy to 
miss, especially if late for departure. I finally caught it when setting the 
V-speeds because they were higher than normal (flaps 1 5 were called 
for in this instance). I have started circling critical items in red on the 
ACCU-LOAD. 

Questionnaire 3 

No responses. 

2.1.12 Training (6) 

Questionnaire 2 

• Current CDU design and or database needs modifications. In NAV 
DATA on the B-757 you can not build your own waypoint. Unlike the 
B-737, this is a great draw back if the waypoints you want are not in 
the database. Most mistakes I see on the line are in reference to this 
one item. 

• FO tried to fly a VOR approach using <the> localizer function. He 
reverted to VOR/LOC logic of his older aircraft. 

• Old computer habits are hard to break. 

Questionnaire 3 

No responses. 
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Error Summary 

The results of the open-ended error question reveals that programming the FMC 
and MCP seem to predominate the pilots’ responses. This is noteworthy 
especially when one considers that there were relatively little reduction in 
programming errors over time (i.e. approximately 25% of the population were still 
making programming errors after a year on the line). One would expect 
numerous errors after transitioning to the line with a tapering off of errors over 
time. This does not appear to be the case with this population in that the 
programming error rate appears to remain steady over time. 

Another interesting trend is the increase in automation error types while flying the 
line. One would expect automation surprises after transitioning the line, and this 
seems to be the case, but an interesting trend is the shift in automation surprise 
responses in Q2 to the reliance on automation responses in Q3. This echoes 
the previously mentioned problem “over use of automation” in some aspects of 
flight (see error type 2.1.3 Automation Q3 on page 16 - “ After arriving at LAX 
terminal area,..." were a pilot became “hypnotized” by the FMC in a last minute 
runway change). 

One area of concern is the failure of energy management during descents. 

There are numerous responses regarding the failure to meet crossing 
restrictions, excessive speeds below 10,000, and altitude busts. Previous 
studies (Wiener, 1989) have also noted a high frequency of altitude “busts” and a 
failure to meeting crossing restrictions and this seems to be the case in this 
study as well. Several pilots attributed the failure to meet crossing restrictions to 
the “clean” nature of the B-757. Meanwhile, other pilots mentioned ATC’s 
tendency to keep the aircraft high until the last minute and then expect a rapid 
descent. In either case, the fact remains that the aircraft is not meeting the 
speed, altitude or crossing restrictions required of certain descents (and ascents 
in some instances). 

Another interesting trend in the error responses is the increase in the action error 
types from Q2 to Q3. The increase in this category is almost exclusively failing 
to select LNAV after being in heading select. This is a curious trend in that there 
are relatively few mentions of failing to activate other functions with the FMC in a 
timely manner. 

Other than the previously mentioned topics, the remaining error rates declined 
significantly from Q2 to Q3 which would be expected as the pilots gain more 
operational experience on the line and familiarize themselves with the aircraft. 
But this is mentioned with a caveat, in that “learning it on the line” may be 
associated with its own set of problems and errors types. 
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3.0 CRM 


Introduction 

As with Wiener’s (1989) previous glass cockpit study, pilots tended to view CRM 
as a workload issue. In addition, the proliferation of automated aircraft and two 
member cockpit crews is premised on the “communication of information” in 
order to maintain situational awareness in the cockpit. This awareness is 
particularly critical during busy flight regimes or when the other crew member is 
busy handling other duties and is “out of the loop.” 

Once again, the CRM open-ended responses are treated in a questionnaire 
specific manner with responses grouped according to either Q2 or Q3 and the 
summary a reflection of both questionnaires. 

3.1 What can you say about crew coordination and procedures in the 757 ? 
Questionnaire 2 

There were 24 specific responses to this open-ended question with the following 
being typical of those who replied: 

• CRM needs to be emphasized all the time in the B-757. (4) 

• The need for teamwork/CRM is very important in the B-757. (2) 

• The design and layout of the flight deck make crew coordination very 
easy and effective. (2) 

• With the abundance of information available in the glass cockpit, it is 
probably more difficult for pilots to ascertain the situational awareness 
of the other pilot crew member. This places a little more pressure on 
the captain to communicate without trampling egos. 

• As long as everyone is operating on the same page, then monitoring is 
good, but when new info is entered on different pages, especially at 
lower altitudes, workload increases. 

• Crew members must interact verbally on what modes are being utilized 
and what the aircraft’s expectations should be. 

• Both pilots must define workload sharing in the advance of flight (capt. 
briefing) in order not to duplicate jobs. 

• The B-757 is a high performance airplane and things happen very 
quickly in a short amount of time. I think it’s important to brief the very 
basics in the event of communication break downs or emergencies. 
There is a need to brief the aircraft’s automation and treat it as a third 
crew member. 

• I like the ways we are operating and conducting procedures in the 
B-757. 

• Excellent procedures in the B-757. Keeps you outside. 
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Questionnaire 3 


Twenty-six pilots responded, with the following being typical : 

• CRM/procedures in the B-757 are good/very good. (10) 

• There is a need to be more of a manager of the automated assets and 
how to use them. (2) 

• The (B-757) requires that we brief each other on what has been 
loaded into the FMC. (2) 

• Standardization on the B-757 is good. (2) 

• It’s difficult to see what the other pilot is inputting when he/she is 
pushing buttons. 

• Coordination in the B-757 requires “buy-in” by both pilots or results in 
the need for one pilot to maintain extra vigilance if the other doesn’t 
understand the system and is unwilling to express it. The procedures 
are designed well, but complacency sets in. 

• CAL’s B-757 CRM and training procedures are the most advanced. 

• Pilots need to treat he automation as a member of the team. 

• One good thing is the sophisticated FMC is almost always right. Crew 
coordination is pretty simple if you both follow SOPs. 

3.2 In what way are they different from previous planes you have flown? 

There were four sub-topics that emerged from the response analysis with the 
following three topics being representative of the responses: (1 ) crew size 
comparisons, (2) procedural and checklist differences, and (3) CRM. 

3.2.1 Crew size comparisons 

Questionnaire 2 

There were 10 replies comparing three-crew member cockpits with two-crew 
member ones with the following being typical responses: 

• I have always been on a 3-man crew and find both of us doing 
something with the FMC. 

• I have been on a 3-man crew for the better part of my time. Switching 
to a 2-man crew changed the way flight was conducted. Once you get 
used to not having the third man onboard, the smoother flight 
progresses. 

• Roles and pecking orders are much more apparent in 3-man cockpits. 

• With an automated cockpit, each pilot must know what the other one is 
doing at all times. In previous planes (3-man), I paid more attention to 
basics. Including looking outside. Now both pilots just sit back and 
monitor what the airplane does. 

• Coming from a 3-man cockpit (B-727), covering all the bases in a 
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2-man crew can be a little busy. 

Questionnaire 3 

There were seven replies which refer to crew size comparisons, the following 

being typical: 

• Most captains, myself included, came off a 3-man crew. That took 
some getting used to. 

• I believe the 3-man crew is a much safer operation. Our CRM and 
procedures are excellent, but situational awareness is unavoidably 
diminished with malfunctions and/or abnormal situations develop. 

• I always have flown 3-man AC. This was my first 2-man AC, and it 
gets busy. But, crew coordination is the same to me. 

• I prefer the 2-man crew when it works as we trained. 

• There is more coordination required in a 2-man crew. All my other 
equipment has been 3-man crews. 

3.2.2 Procedures, checklists, and CRM comparisons 

Questionnaire 2 

There were 28 similar response with the following being typical: 

• No taxi checklist. (2) 

• Average CRM. (2) 

• Cockpit flow as <compared> to specific checklists. I prefer checklists. 

• The B-727 has a very long drawn out series of checklists with a lot of 
switches and buttons, where the B-757 checklists are short and 
concise with system checks being short and quick as well. 

• The simplicity of checklists. 

• I came from the B-737-300/500 and there is more emphasis on “hitting 
the box." 

• Each pilot has to be aware of the other pilot’s input into the FMC and 
other pilot’s thought process. 

• CRM and procedures are for the most part SOP. The main difference 
is higher pre-flight workload and closer coordination for in-flight 
programming. 

• Crew coordination is about the same. 

• Better coordination and procedures than any plane I’ve flown. 

• <l> came from the A-300 and CRM is adequate. 

• CRM works very well and is stressed more than other planes. 
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Questionnaire 3 


There were 16 similar responses referring to checklists, CRM, and procedures: 

• I prefer the short checklists and flows. (2) 

• Checklists are simplified and there is no taxi check. (2) 

• Same procedures. (2) 

• I had previously spent 9 years on the B-737-300/500. The transition to 
the B-757 was fairly easy. The biggest improvement was shortened 
checklists and cockpit layout (most noticeably the HSI mode selector). 

• Crew coordination is much more important in the B-757 than any 
current aircraft in the fleet. 

• There is more coordination required on the B-757. 

• CRM is stressed by CAL. 

• They (procedures) don’t seem that much different than the CRM 
procedures for other airplanes. It is possible for the PNF to get out of 
the loop. 

• Crew coordination is better (in the B-757). 

• B-757 crews seems to work more as a team. Still have “hot-shots” 
who are always on the computer and push buttons too fast. 

• The (B-757) requires better communication between crew members as 
info can be loaded in the CDU without the other crew member knowing 
about it. 

• More CRM in the B-757, some of it in relation to the increased capacity 
of the airplane. 

• Crew coordination is simpler because it's easier. 

3.4 What areas can use improvement? 

Questionnaire 2 

There were 34 recommendations made by the pilots on how to improve 
procedures, checklists, and CRM. The typical recommendation responses were 
as follows: 

• We need a clearer separation of CDU/MCP duties. (4) 

• More emphasis on standard procedure/checklist usage. (4) 

• Would like a taxi checklist with the flight controls check done at that 
time. (2) 

• There needs to be additional care taken in observing and cross- 
checking the programming of the FMC. (2) 

• Have the PF make the CDU entries like on the B-737-300/500. Why 
not have consistency across all aircraft types? (2) 
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• Procedures for setting the ALT ALT/warning on the MCP should be 
changed. I think that if the PNF always made the ALT change (like A- 
300), there would be less room for error. 

• Would like to see some reference to flaps on a checklist, possibly on 
the after start check. 

• I still find some people who are resisting procedures to call for 
CDU/FMC/MCP functions when they are the pilot flying, especially in 
the terminal environment. 

• Duties regarding who should program the FMC when PF or PNF. 

• Most FOs do not verbally announce changes in the MCP altitude 
settings. The flight manual mandates doing so and I think this is a 
training or awareness item. 

• I am all for short checklists, but it bothers me that our take-off and 
landing checklists leave things to be done after the checklist is 
complete. We should be at our final flap setting before calling for the 
landing checklist instead of doing the checklist at 20 degrees. 

• The company has gone overboard to keep checklists short. The flaps 
not being on the after start checklist is a crime given that there is no 
taxi checklist. They are part of the “flow”, but don’t appear on a 
checklist until before takeoff, when it’s a little too late to be lowering 
the forgotten flaps. I’ve gone all the way to the runway with the flaps 
up, neither pilot noticing. 

• It’s hard communicating with all the flight attendants (we usually 
carry 5). 

• Works fine as is, no improvements needed. (3) 

Questionnaire 3 

There were 19 suggestions from pilots with the following being typical responses: 

• A more specific defined policy of one pilot programming the FMC. (3) 

• Need to have better CDU input verbalization, especially in the 
terminal. (2) 

• More checklist discipline: calling, responding, and timely execution. (2) 

• More emphasis on less FMC/CDU below 10,000. 

• The approach checklist is used at a time when the cockpit workload is 
very high. 

• There is a need for PNF to monitor FMC inputs by PF, in order to 
verify and backup. There is a tendency by some captains who are 
PNF to make FMC inputs. This can be confusing to the PF. 

• I am quite alarmed that while the B-757 procedures are instructed 
during training, there are many cross-over procedures that creep in to 
line operations “well, that’s the way we always did it on the Airbus.” We 
need enforcement of the concept that this is the B-757. 
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• Below 18,000’, I’d like to have both crew members in the loop and not 
talking on the radio to company and doing maintenance write-ups. 

• I believe that calling for the flight guidance changes helps keep the 
non-flying pilot “in-the-loop.” Yet, there seems to be tremendous 
reluctance to do so. It embarrasses some pilots to verbalize 
commands they are used to performing themselves. 

• There needs to be an improvement in interpersonal skills. Some pilots 
can not relate to their fellow man and equally as poorly with a know-it- 
all computer. 

3.5 Miscellaneous CRM responses 

Questionnaire 2 

In questionnaire 2 there were nine responses which were not easily categorized, 
as a result, they were placed in this miscellaneous response category: 

• I appreciate the active teaching/applications of CRM. 

• CRM has always been good at CAL in my experience. 

• I am a proponent of strong CRM. I stress it, set the environment for it, 
and it seems to work. This requires an open, receptive and forgiving 
captain for excellent CRM to work. 

• Complacency could become a problem if everything is loaded properly 
and things work great. Success and ease brings complacency. 

• At this carrier, I find CRM to be very good. 

• CRM is a very good tool to improve cockpit inter-relationships. 

CRM summary 

There appears to be an underlying theme in the pilot’s responses which reflects 
a need to effectively monitor, communicate and manage information on the flight 
deck. In fact, one pilot referred to the automation as a “team member” and 
suggested treating it accordingly. However, the abundance of information could 
“swamp” a recently transitioned pilot or present difficulties ascertaining another 
crew-member’s awareness of the current flight regime. This was alluded to in 
several statements regarding further clarification on duties and procedures for 
inputs into the FMC/CDU. 

A topic which received many comments was the 2-person vs. 3-person cockpit. 
This still seems to be a prevalent topic among some pilots even though the 2- 
person cockpit has been in service for several decades. It appears that most 
pilots transitioning from a 3-person crew acclimate to the new situation fairly 
easily, however, they are still a few pilots who are opposed to the loss of the 
flight engineer and, in fact, returned to a 3 seat aircraft based solely on that fact 
(see section 5.0 For those who have left the 757). 
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As for procedures and checklists, most pilots preferred the shortened checklists 
and flow patterns associated with the B-757. However, there was mention of 
“forgetting” to set the flaps before arriving at the runway thresh-hold for take-off 
and this raises some concern. In general, most of the pilots endorsed the B-757 
procedures, checklists and Continental’s strong CRM approach associated with 
this aircraft. 


4.0 Workload 


4.1 How would you compare the overall workload in the 757 compared to 
your previous plane? 


Workload Compared to Previous Plane 

Total = (Q2 + Q3) 

4.1.1 Much less than previous plane 

16 = 6+10 

4.1.2 Less than previous plane 

32 = 15 + 17 

4.1.3 About the same as previous plane 

26 = 13 + 13 

4.1.4 More than previous plane 

12 = 8 + 4 

4.1.5 Much more than previous plane 

7 = 3+4 

4.1.6 Shifted or different from previous plane 

3 = 0 + 3 


General Comments on Workload 

Questionnaire 2 

ATC and effect on workload 

• Last minute changes causes the workload to increase at critical 
times. (3) 

• ATC has a tendency to keep you high expecting a rapid descent and 
or speed reduction which is hard to do in a B-757. 

• Under normal conditions it is less. When ATC makes changes that 
were not programmed in the FMC, then it becomes more workload. 

• ATC calls at inopportune times. 

• Need to allow for published FMS arrivals. Too often ATC cancels 
them. 

• ATC is always changing to their benefit and increases our workload. 

• It’s helpful when ATC understands how the automation works and 
clears us with that in mind. 

• ATC is inconsistent. 

• ATC interferes with the automation too often. 
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• Once ATC changes the flight plan the FMC needs to be changed and 
this increases workload. 

• Some controllers at ATC still seem unaware of the aircraft’s/crew s 
capabilities and do not make full use of what we can do. Sometimes 
they even become argumentative when we try <to> help. 

Phase of Flight and Workload 

• The majority of workload on the B-757 is prior to departure and after 
landing. 

• Pre- and post-flight workload is higher. In flight workload is lighter. 

• The overall workload seems to greatly reduced during most phases. 
However, the loading of the flight computer seems to take a lot of time 
and concentration. I am referring to the pre-flight phase. 

• Overall workload in pre-departure, post arrival and most terminal 
environments is considerable higher. Especially in a fluid ATC 
environment. 

• The loading procedure during preflight is laborious and time 
consuming. To do an effective loading of the FMC takes a good 10 - 
1 5 minutes with numerous opportunities for errors. 

• The workload is far greater during the departure and arrival phases of 
flight for FMS aircraft than for non-FMS ones. 

• <Workload is> much greater except at cruise. 

• The workload at the start of the flight (in chocks) is the highest I’ve 
experienced. However, once airborne, the load is very low and it’s 
easy to lapse into complacency. The need for vigilance is even greater 
for that reason. One note: CAT III approaches and autoland are 
probably the place where the difference in workload is the most 
pronounced. 

• Higher preflight <workload> is bad in B-757. 

• An increased workload just after take off.. After that, much easier. 

• The important thing, however, is when it <workload> is less. I do most 
of my work before the flight even begins. Then I am able to better 
watch and/or manage the balance of the flight. 

• I felt that one area of concern that I had, even after IOE, was descent 
management. In fact, I see the same thing in many of the new pilots 
coming on the line. The problem I am referring to is not “planning’’ a 
descent, but executing it safely and efficiently with the flight guidance. 

• Descents are unpredictable in the B-757 system. 

Workload and long haul operations 

• On B-757 there is less workload on all phases of flight except actual 
ETOPS and long-range NAV (look at ETOPS checklist). 

• Very high! A very high workload is especially prevalent during 
international operations. 
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• I just flew to Manchester. The trip over was fine. Coming back we had 
everything changed en route (track, ETOPS altn 3 times, etc.). Two 
pilots were too busy to do all of this. An IRO was needed badly. If any 
abnormals would have happened, we would have been so busy that 
“flying or monitoring the A/C” would have been tough. 

Workload and lack of a flight engineer 

• More workload due to the absence of a flight engineer. The pilots have 
to pick up the company and cabin workload. 

• Much greater - 1 came from 3 pilot A/C to 2 pilot A/C but automation 
lessens the workload. 

• I was previously on a 3-man A/C (DC-IO)and there is no comparison. 
Work was easier with the S/O doing his job. 

• I did not assume !4 of the S/O’s workload, but from 50% to 1 00% 
additional. When an emergency requires radio coordination with 
company personnel, one crew member's workload increases many 
fold. Being a single engine, single seat trained pilot, I am somewhat 
accustomed to the workload. However, when the F/O has to confer 
with the capt. for information and/or decisions, the capt. can become 
overloaded. A third crew member would be an enhancement in certain 
environments. 

• Workload is slightly higher, but mostly during pre-flight (no second 
officer), but nothing unmanageable. 

• Increased workload because of a 2-man cockpit. 

• Workload is automated and it helps cover a lot of items that were 
manual by the second officer. 

• The second officer is helpful in monitoring approaches, reading 
checklists, and obtaining ATIS and company data. 

Workload and comparison to previous plane 

• B-757 workload is about the same as B-737-300/500. (3) 

• The whole cockpit of the B-757 is much busier than the old airplane 
(DC-10) with a 3-man crew, but that’s not the only reason. The 
automation has to be programmed and that takes time, knowledge, 
and practice. The FMC is nice for planning - if you have the time. 

• Much less than B-727 compared to flying and following clearances. 

• Easier than the B-737-300. 

• Workload on the B-757 is greatly reduced versus the B-737-300/500 
because of our long route segments versus the short ones on the 
B-737. 

• The overall workload is certainly no more than the MD-80 or B-727, 
and, in most situations, considerably less. I think the workload is 
already easy to manage. 

• I find the workload slightly less than the B-737-300/500. 

• Much less than the B-727 and the B-737 (due to the long legs). 
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• The B-757 is not any busier than any other A/C I have flown. 

• Workload on the B-757 is greater than on the older generation of 
aircraft. For example, on the DC-9, in order to plan a descent, I would 
figure the distance needed to cross a point at a certain altitude and the 
rate of descent or airspeed would be established and both monitored. 
On the B-757, 1 still have to figure the descent, then check the 
computer and if both agree, then start the descent at the correct time 
while making sure it holds the speed, autothrottles are coming up to try 
and hold the speed but then reduce for the descent rate, and the 
green arc is correctly displaying the situation. In this instance, it’s 
easier to fly the plane than the "computer.” 

• It is higher than the DC-1 0. 

• My previous plane was the DC-10 which had 3 pilots thus spreading 
the workload thinner, but I find the B-757 the easiest plane I’ve ever 
flown both in terms of workload and flighty characteristics. 

• The B-757 and B-737-300 are equal during gate departure, takeoff, 
etc., but the B-757 is significantly higher during descent due to its 
tendency to get fast and high on descents. This causes much greater 
use of speed brakes, mode switching, button punching of the 

FMS, etc. 

• The overall workload in the B-757 has increased substantially in the 
critical phases of flight, compared to the previous A/C I flew (B-747, B- 
737, B-727, DC-10, and DC-9/MD-80). An emergency arising during a 
critical phase of flight can overwhelm even a well trained and 
experienced crew. Your average crew could be overloaded or 
distracted with mush less. This advanced twin engine, two pilot aircraft 
has now been flying for years and has managed to stay out of the 
headlines, perhaps due to its relatively small numbers. 

Workload and gaining experience with the B-757 

• The workload is now becoming less as I gain more experience. 

• Right now with just a little over 100 hours, the workload is probably 
more than my previous plane, but I feel that with another 6 months or 
so, I <think> it will be the same if not less. 

• As I become more familiar in the B-757, workload appears to be less 
and less. 

Availability of information and situational awareness 

• More information is available to me in the B-757. Workload is no more 
or less, but I have more options and displays from which to choose. 

• The workload is greatly reduced in having the ability to evaluate and 
improve situational awareness. 

• Workload is much lower yet have more information to help make good 
management decisions. 
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Questionnaire 3 


ATC and effect on workload 

• ATC has not fully upgraded equipment and is unable to make full use of 
today’s automation. It <B-757> of course is restricted by combination of 
all A/C. Maybe 20-30 years from now when most A/C will be auto? 

• In high density areas, the call in to Departure Control is right at our 
flap/power transition point. If ATC could live with just the call sign 
(company and flight number), we could be better at flying, looking, 
transitioning and being more coordinated/safer. At EWR (for example) 
we could auto switch to Departure Control at 500 ft (or some agreed on 
altitude). The point is that you’ve got radio calls, flaps to change, 
headings to dial, altitude to set (all in 5 seconds). Parcel it out. 

• The problems with ATC, (LAX and SFO) it’s hard to use all the 
automation, because airports like these are too busy. Speed up, slow 
down - stop your descent or climb so you’re back to a basic A/C. 

• ATC in arrivals and departures are the biggest workload. We need a 
better ATC system to manage ARR and DEP. 

• ATC procedures have become a real pain. I don’t believe the changes 
are necessary and I believe that controllers have an agenda that they 
are promoting through their “erratic changes” in traffic control. 

Phase of flight and workload 

• Too much company and outside interference during block preflight 
duties and check list. 

• I believe the workload to be higher I the B-757 during some phases of 
flight i.e. preparing A/C for flight (on ground) and at altitude prior to 
descent (especially when changes occur to expected clearances, SIDs, 
runway changes). However, these normally do not occur during critical 
phases of flight - once programmed, the automated cockpit gives you so 
much more information during the critical phases of flight i.e. T.O., 
approach, landing. 

• Higher workload during preflight and in general there is less workload in 
flight. However there is an increase in workload with the B-757 in the 
terminal area when there is a change in runway or instrument approach. 
At high density airport such as LAX, a greater EFIS workload reduces 
ability to look for other airport traffic. The slowness of the FMC 
contributes greatly to the problem computer speed should be increased. 

• Ground duties present a much higher workload than other equipment 
such as B-727, B-737-1 00/200, DC-9 & MD-80 and close to DC-10. The 
pilots need to arrive at the aircraft to set up much earlier than the 
Jurassic jets. Thirty min show at operations at outstations is 
unacceptable. The duty day should be 1 hour show both at the start of 
the paring as well as downline. Competent crews that are ahead of the 
aircraft and not in <cabin> row 23 should have no problem managing the 
operations. If ATC can utilize FMS arrivals and sequence other A/C 
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along those routes radio transmission could be cut substantially. This 
would provide a much safer environment. 

• More in chocks, less in taxi, same on T/O, less in climb and cruise, more 
on descent, less on taxi, more in chocks. 

Workload and long haul operations 

• More workload on NATRAC-ETOPS RTES <when compared to a 
DC-10>. 

Workload and lack of a flight engineer 

• It’s much higher, naturally, since there is one less person in the cockpit. 
Normal arrival at gate communication procedures should be handled via 
ACARS up/down link. Having to make several radio calls and monitor 
frequencies other than TWR/GND is inherently error prone in a 2-man 
cockpit. 

• Much greater workload on B-757. Two-man cockpit in today’s arena, 
with all required radio work, especially if something goes away, i.e. 
divert, emergency, etc., overloads crew to a dangerous level. I do not 
have a solution, except to bring back the flight engineer. 

• I was a F/O on B-727. My workload increased greatly! 2-man A/C 
should never have been approved to begin with. Too much inside and 
not enough outside! 

• <Workload> is less than B-727 even with 2-man vs. 3-man crew. 

• Generally less (than the B-747). 

• The previous A/C had a flight engineer, <now> we have the FMS. 

• Having the flight engineer is a wonderful luxury. However, in certain 
environments, must notably terminal arrivals and departures, a last 
minute clearance change (change of rwy, route, or approach) can leave 
me feeling overwhelmed. In these cases I generally revert to raw data 
automated operation (heading select, etc.). When time and workload 
allows I then bring the FMC up-to-date again. 

• Workload from a 3-man aircraft to a 2-man aircraft increases no matter 
what automation is added. 

• A third crew member would be helpful to accomplish all “clerical” 
responsibilities. 

• Much easier without the F.E. I enjoy doing the cockpit prep with all the 
datalink hookups and not having to forget to get a word in on a 
congested frequency. Captain can easily and efficiently validate the 
accuracy of the data. 

Workload and comparisons to previous plane 

• B-737-300/500 <is> about the same. (3) 

• The initial loading of the computers takes more time and rechecking 
than the previous A/C (A-300). The uplinking of data is very helpful 
when it works. 
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• I feel the B-757 workload is easier than previous A/C due to glass 
cockpit. 

• Previous plane A-300. How about <a difference in> night & day? 

• I have flown every airplane type in the Continental fleet. The B-757 has 
without a doubt, the lowest overall workload. 

• Much less than B-727 - once you master <the> FMC. 

• The workload is much less than an MD-80 or DC-10. 

• Workload between DC-1 0 and B-757 is very close to the same - the big 
difference is that EFIS in the B-757 results in better situational 
awareness. 

• Higher workload than B-727 for arr - dep. 

• The workload is higher in the B-757 than the B-737-300 because of the 
FMS design. Subtle mode switches, like VNAV Path to VNAV Speed, 
should have never been FAA approved. This increases workload by 
requiring a fixation on VNAV performance - to the detriment of 
everything else that is happening. 

• Workload? What workload? In comparison to the MD-80 and DC-9 
there is less to do i.e. Accuload via printer, PDC via printer, but A/C 
requires a different approach and method of conducting yourself. They 
also fly different stage lengths and frequencies. The B-757 has to be 
one of the least stressful aircraft, operating, usually one or two legs per 
day. 

Workload and gaining experience with the B-757 

• The first year much heavier. The 2nd year - the same; starting <the> 

3 rd Year - easier. 

• When I first checked out on the B-757, 1 found the workload to be quite 
high! I was not accustomed to having to do so many additional duties. 

I now find the work load to be much lower. Automation helps and my 
flow patterns are refined. 

• Until recently the workload in the B-757 was about the same as it was 
on previous aircraft. The difference was one of workload distribution. 

Availability of information and situational awareness 

• Much lower <workload>, the automation allows you to stay much further 
ahead of the airplane! 

• The workload on the B-757 is much less. The navigation tasks are 
greatly reduced and there is a wealth of information about the flight that 
can be accessed. If there is a drawback it is that the system is so nice it 
breeds complacency. 

• The map display really makes the flight progress nicely. I like the 
display system with the airport and runways shown on <the> screen. 

• Workload during routing tasks in a low stress environment is greatly 
reduced. However, I am not absolutely sure that the workload is 
reduced during high stress situations, i.e. approaches or complicated 
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arrival procedures particularly when a modification is introduced by ATC. 
Personally I find the workload to be increased because I must decide 
upon the manner in which the change <takes place>. I may have 
several options available to me and that in itself complicates the 
procedure. With the old “stream gauges” you tuned it, dialed it and flew 
it. Simple. And both pilots knew what procedures was taking place. 

• Quicker, more accurate assessment of overall situations as so much 
information is available. 

• The EFIS helps my situational awareness, which makes the overall 
workload on the B-757 even less. 

4.2 Please mention anything you feel should be changed to help you 
manage workload ( procedures , ATC, training etc?) 

Questionnaire 2 

• No changes needed (so stated). (3) 

• It would be nice if more routes could be stored in the database. (2) 

• Having to reset the altitude to an intermediate altitude on the MCP on 
a profile descent defeats the automation and increases unnecessary 
workload levels. The intermediate altitude showing on the moving map, 
the legs page, and the descent page are all an adequate check that 
the AC will not bust an intermediate crossing restriction. 

• We should have a sterile environment @ 20 minutes prior to departure 
and no cabin or company contacts or duties after descending for the 
purposes of landing. 

• More defined procedures as to capt/F.O. responsibilities for cockpit 
setup (data loading, panel setup, etc.). <Also,> Orange County ATC - 
very busy and heavy workload. ATC is calling at the busiest time 
(tower to departure handoff). This occurs at the same time as a very 
busy FMS departure. Less talk would help. 

• It would help all types of aircraft if the WX sequences would report 
runways in use at the reporting airport or if we could get ATIS farther 
away from the destination (or alternates en route). 

• ATC procedures for FMS aircraft should be modified to minimize the 
time required for pilots to direct the FMC. This will probably occur 
naturally as the majority of airborne systems become FMS 
controlled. The problem exists due to a mix of FMS and non-FMS A/C 
in today’s ATC environment. Arrival and departure procedures could 
be modified to simplify the phase of flight requirements for all A/C. 

• The company needs to standardize databases on both the “glass “ A/C 
(B-737-300/500 and B-757). 

• Some of the preflight CDU entries could be automated or eliminated. 
Examples: (1 ) descent winds have to be input twice, (2) engine oil 
quantity, (3) DFDMU data, (4) origin and destination (route page 1), 
and (5) shutdown fuel. 



• No company business below 1 8,000’ but call company when on the 
ground and taxing in. 

• Complete and maximum utilization of datalink capabilities in order to 
ensure 100% concentration in monitoring the aircraft during critical 
phases of flight (i.e., descent in dynamic traffic environment, low 
altitude transitions, etc.). 

• Maybe an updated ATC system with less power failures or equipment 
problems would help to bring down the workload of both the pilots and 
air traffic controllers . 

• (1 ) Speed up the computer, and (2) an extension line to ILS final 
should not have to be manually input. This would eliminate one step 
which is heads down time in the terminal area. 

• The software should be examined to see why the airplane always gets 
fast and high on descents. During my first 100 hours, I have only had 
one descent using VNAV path which did not require a significant 
application of speed brakes. This was not the case on the B-737-300. 

• The only thing I’d like to see is a redesign of ATC to cater to the 
automated aircraft and not keep messing up our plans. I believe they 
are working on it but time will tell. After all, most A/C are FMC 
equipped these days. 

Questionnaire 3 

• The slowness of the FMC contributes greatly to the problem <of high 
workloads Computer speed should be increased. (2) 

• No changes needed (so stated). (2) 

• Make it possible to retain <past> waypoints on the CRT for a while. 

• Too much company and outside interference during block preflight 
duties and check list. 

• Reduce the flying workload for long haul so more situational 
awareness can be given to terminal areas. 

• ATC should get up to speed with the rest of the world with their 
computers, training, facilities, etc. 

• I do not have a solution <for increased workload>, except to bring back 
the flight engineer. 

• Approach checklist is a nuisance - should be changed. 

• Training should emphasize MCP first below 10,000 ft. 

• ATC realizing and using automation. 

• (1 ) Center the HSI display and TCAS. <Also,> both now wipe out 
anything under you or immediately behind you. (2) GPS updates to 
position (both 1 and 2 would have helped the B-757 crew in South 
America <Cali, Colombia accident>). 

• Automated ATIS info in ACARS would help. 
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• The biggest restraint I feel out there is ATC. I calculate the optimum 
descent point to the nano-second and ATC starts me down 100 miles 
early, or late. I don’t need an FMC for that. 

• I would change the chiming of the no smoke sign at 10,000 ft to 
18,000 ft. At 10,000 I feel the crew is to busy i.e. slow to 250kts, ATC, 
traffic watch. 

Workload summary 

As mentioned in the previous section (3.0 CRM), the 2-pilot vs. 3-pilot flight deck 
is a prevalent topic among some pilots. Some of the reasons given for 
maintaining the second officer are for safety reasons (i.e. “extra" pair of eyes and 
for emergency situations), but also for routine duties such as company 
communications and paperwork. In fact, one respondent mentioned the need 
for a “secretary" instead of a flight engineer to handle “clerical” duties. 

The topic of ATC and workload received many comments and were almost 
unanimously negative in tone. There seems to be a perception that ATC is a 
major factor in some high workload situations, especially in arrivals, departures, 
and failing to make full use of the B-757’s automation. It is an unfortunate 
situation when pilot’s have a negative perception of ATC with respect to 
automation, workload, and perceive ATC as acting in their own interest. In 
addition, international operations (ETOPS) also received unanimously negative 
comments due to the high workload involved. 

When comparing workload to their previous planes, a majority of pilots perceived 
the B-757 as having less or the same amount of workload as before. There 
were a few comments stating that the B-757 was more workload than their prior 
plane. However, one must take into consideration whether or not there was a 
shift or difference in workload when compared to their prior plane. Although only 
three pilots stated that there was a shift or difference in workload, many pilots 
commented that the workload increased during pre-departure and post-arrival, 
and decreased in cruise. The workload issue must also be approached with a 
caveat in that many of the B-757 flights were long-haul flights instead of shorter 
legs associated with some glass aircraft previously flown (B-737-300/500, etc.). 


5.0 For those who have left the 757 (questionnaire 3 only) 

5.1 After you left 757 and went to another aircraft, what was your 

reaction? 

5.2 What did you miss about the 757 avionics and automation? 

5.3 What did you like better about the older technology plane? 

(Please state the seat and plane you went to). 
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• (Unknown, unknown). I like the 3-man cockpit. Also, at Continental, 
the B-757 trip schedules in the Houston base are terrible. For the 
same pay, the lines on the B-737 are far superior, I miss the B-757 
and ILS automation. But I don’t miss the schedules. 

• (Captain, MD-80). I left the B-757 training program after completing 
ground school training. The avionics and automation surpassed both 
the DC-10 and B-747 I had previously flown as first officer. A 
company bid allowed me to move to MD-80 captain, my present 
aircraft position. 

• (Captain, B-727). <l> will start training on B-727 March 5, 1996. <l> 
will miss B-757 plane but not the schedules. 

• (Captain, MD-80). I felt it was a step down in technology. All the bells 
and whistles. The glass cockpit, the features in the database. The 
older plane works better in today’s environment because of the speed 
of technology in the A/C is exactly 20 years behind and exactly where 
today’s ATC environment is. 

• (Captain, B-737-300). As I stated earlier, the B-737-500 is more 
advanced NAV-wise than the B-757. I do miss some of the system 
automation and the EICAS system on the B-757 however. 

• (Captain, B-727). The only thing I miss from the B-757 is the map 
mode. Favorite items of B-727: the 3-man crew of the B-727. 

- ability to hand fly much more. 

- more “in the loop” - a pilot again instead of systems monitor. 

• (Captain, DC-9). First I went from F/O on the B-757 to F/O on the DC- 
1 0. That wasn’t too bad of a transition. Then from DC-1 0 F/O to DC-9 
Captain. Then I saw the changes from automation. No auto throttle, 
no full auto pilot. No VNAV or LNAV. Life was good. Now I work. 

• (Captain, B-737). I have flown the B-737-300 and B-757 for 10 years. 

I miss the automation. 

• (Captain, DC-10). Moving to the DC-10 was like meeting an old friend. 
I liked the B-757 simplicity of items to be checked before flight and 
checklists. I miss the map on the B-757. The two best features about 
the DC-10 are: three crew members <and> international FAR’s 
concerning flight time limitations. Allows you to have much more time 
off. The domestic trips on the B-757 are terrible due to ridiculous FAR 
limitations. 

• (First officer, B-727). I bid off the B-757 because of the automation 
and bad trips. The only thing I really miss about the B-757 is 
automation is the printer. The older technology is real flying and it’s 
fun. The older technology makes you a better pilot, by hand flying and 
using your brain. 

• (Captain, DC-10). I missed the moving map and the auto departure 
features in LNAV and VNAV for complicated departure and arrivals. 

• (Captain, DC-10). The B-757 is the best aircraft I have ever flown. 

The glass display was easy to program and maintain situation 
awareness. I have been off the B-757 for 8 months and have probably 
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forgot about small problems that I encountered. Navigation 
comparison of B-757 vs. DC-10, 1 hate the “old way” of getting there. 
So much more info is quickly accessible in the B-757; I believe it to be 
the safer way. 

• (First Officer, DC-10). Miss the FMC and map on screen. I like the 
3-man crew in DC-10. 

• (Captain, DC-10). The B-757 instrumentation results in much superior 
in situational awareness ( a quick glance at the Map and you know 
exactly where the landing runway is and your orientation to it). The 
DC-10 requires at least two mental maneuvers to change the existing 
instrumentation into an “estimate” of where the landing runway is - and 
may even require a frequency change on a VOR or ILS. I changed the 
A/C because of better schedules (more days off) on the DC-10. 

• (First officer, DC-10). I miss everything. I wish the automation was in 
the DC-10 or B-747-1 00/200. The reason I left was <that> the 
company’s workload, days off, and limited crew rest at destination was 
undesirable. Domestic flying is not a challenge and boring so I 
switched. My dream would be a B-747-400. 

• (Captain, DC-10). I am planning on returning to the DC-10 ASAP for 
the following reasons: (1) 3-man cockpit, (2) 3-man cockpit, and 

(3) 3-man cockpit. I would very much would like to have “some” of the 
technology to augment or enhance situational awareness/auto land 
capabilities and not to replace the resources necessary to operate 
airlines safely. 

Summary, for those who left the B-757. 

Seventeen pilots responded to these questions and most seemed to miss the 
automation but a few respondents preferred the older “steam gauge” cockpits 
and the three-person flight deck. Several comments were made regarding the 
dislike for the B-757 flight schedules and left the aircraft for this reason (ex. 
“bad schedules”, “bad trips”, etc.). Of the 17 respondents, 11 transferred to the 
captain’s seat in other aircraft. It is understandable to desire a captain’s position 
and one may suspect this may have influenced a pilot's desire to leave the B- 
757, especially if one has never held a pilot in command position previously. 


D. CONCLUSION 


In conclusion, Continental’s training program for the B-757 received much praise 
from the participants. This is a commendable achievement for the training 
department and all those involved who developed and implemented the program, 
especially considering the B-757 was a new aircraft for Continental at the time. 
Although there were numerous complaints and suggestions put forth on specific 
topics, not one pilot was dissatisfied with the training program in general. This 
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does not imply that the training program would not benefit from refinement or 
incorporating some of the suggestions put forth by the pilots in this chapter. 
Some of the suggestions and comments put forth by the pilots are quite apropos 
and worthy of further examination for their potential applicability to CAl’s training 
program as well as other airline’s training programs as well. 
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VII. 


CREW RESOURCE MANAGEMENT AND 
HUMAN-CENTERED AUTOMATION TRAINING 


Our company doesn’t teach CRM. We teach pilots to fly and to manage 
checklists, communication, ATC, and systems. CRM is an integrated part, the 
parts are inseparable. We integrate CRM into everything we do, every minute 
that the crew members function together as a crew. 


— B-757 flight instructor 


A. HISTORY OF CRM AT CONTINENTAL 


Continental was early on the scene of cockpit resource management (or "crew" 
resource management, which is presently more in style due to its generality). The 
programs today, and the general acceptance of CRM training can be traced back to the 
pioneering work of Captain Frank Tullo. By the end of the 70s the movement was 
underway at several U.S. airlines (Cooper, White, and Lauber, 1979). By 1981 United 
Airlines had its own program (CLR - Command, Leadership, and Resource 
Management) in position, and offered it for sale to other carriers. As early as 1977 
Texas International developed a simulator program that today would be called LOFT. 

Tullo and others at Continental watched the United experience with great interest. He 
once gave the following example (personal conversation with first author, 1993) of the 
sort of thing that bothered him. A Boeing 727 taking off from Phoenix lost an engine at 
Vr. They managed to get into the air, and at 200 AGL the crew ran the engine failure 
checklist. Tullo stated that the crew “got nothing in return” for their efforts. “They could 
have climbed to 800 feet, accelerated, cleaned up the airplane, and then run the 
checklist. Perhaps United had the right idea: some non-technical training that would 
emphasize teamwork and communication skills might be the answer,” he stated. 

Following the strike in 1983 (see Chapter I), Tullo was named lead flight instructor on 
the 727, and took the opportunity to press for a program in what would later be called 
CRM. The first course, called Crew Coordination Concepts (CCC), and later called 
Phase I, was something of a hodge podge of personality and small group dynamics, 
communications, the theories of the situation of Professor Lee Bolman of Yale, and the 
Blake-Mouton grid (see Foushee and Helmreich, 1988, pp. 201-203). The grid would 
soon enjoy great popularity in the years to come. Continental management supported 
the proposal, resulting in a two-day course which was case oriented, stressing as raw 
materials carefully selected accident reports. These are the basis of many of the 
programs around the world today. 
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In 1986 the company felt that the course was running so smoothly that it could be 
marketed to other airlines. And in the 1980's there was a plethora of accidents that 
would make excellent examples of the need for improved CRM in the cockpit (see 
chapters in Wiener, Kanki, and Helmreich, 1993, especially the chapter by Kayten). 
Management made a long-term commitment by appointing Tullo as manager of the 
CRM program. The CRM program today runs under the guidance of a committee of 
pilots, and a full-time non-pilot administrator. In 1988 a Phase II program (one day) 
was installed. Phase II dealt primarily with decision making. The next (Phase III) 
program stressed teamwork and leadership. Work began in 1987 on recurrent LOFTs 
for the many fleets of Continental. The LOFTs were strictly recurrent exercises, not 
“CRM LOFTS” which were gaining in popularity, and which might soon be required in 
order to meet AQP requirements (FAA, 1998; General Accounting Office [GAO], 1997). 
For an excellent discussion of LOFT design, philosophy, and execution see R. Butler, 

1 993 and Orlady, 1 994. 

Today the programs are still influenced by Captain Tullo's early work. By 1 988 
Continental had filed applications for AQP programs for most of its fleets. The CRM 
programs were one day for existing pilots and two days for new hires. By the 1 990's 
most pilots in the U.S. had been through some kind of CRM program. The emphasis 
was changing from communications and personality dynamics to newer concepts such 
as error management, decision making, and situational awareness. A tidal wave was 
sweeping though airlines’ fleets. It was called the “glass cockpit” (Wiener, 1989). CRM 
managers fell heir to the unexplored problem of training for the high technology aircraft 
(Wiener, 1993). The frustrations of the early days of the glass airplane have been 
documented in Chapter II. It was concern over these problems that brought the current 
(NASA/Continental) study into being. For a detailed discussion of integrating CRM into 
an AQP environment, see FAA (1998), Chapter 9 (Crew Resource Management), and 
McKinna, (1996a). 


B. MODERN APPROACHES TO CRM 


The post-Tullo era at Continental saw a change of emphasis. The key word was 
integrated. The quotation at the beginning of this chapter captures the meaning of the 
word when applied to CRM training. Integrated CRM training simply means that CRM 
was no longer taught as a subject matter in itself, a skill which the airman would master 
in its abstract form, and might some day have the opportunity to apply it to his work. 
Instead the subject matter would be taught with CRM woven into each topic. For 
example, in the B-757 transition training, the instructor would discuss not only the 
maneuver and the technical procedures to be employed, but would include as well the 
callouts and briefings required, and show the students how to combine technical skills 
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with CRM skills. The communication, leadership, division of duties, etc. would be 
taught as the ingredients of the maneuver, not separate functions to be performed by 
the pilot. 

Influences 


There were two influential agents that led to the move toward integration. The first 
was the demands of AQP (FAA, 1998). Compliance with AQP required instructors to 
teach integrated maneuvers rather than procedures, to grade the CRM aspects of the 
maneuvers as strenuously as he might grade the execution of procedures, and to grade 
the crew, not individual crew members. This in itself was a radical notion, that grades 
would be handed out to the entire crew , rather than focussing on the performance on 
individuals within the crew. 



FIGURE VIM. Attitude toward company CRM training. 


The second influence was the instructors that were encountered by CAL s initial 
training cadre at Boeing’s training center. Apparently one Boeing instructor had a 
tremendous impact on the Continental instructors: he demonstrated how CRM could be 
integrated with technical flight training. The training cadre was determined to bring 
both the spirit and the methods of integrated training back to Houston to become 
training doctrine at the airline. This turned out to be particularly helpful in the glass 
cockpits (at that time B-757 and 737-500). As one instructor put it, “The more tools you 
give the pilot, the more CRM is needed.” In previous field studies, Wiener (1985b, 
1989) had said essentially the same thing, that high technology cockpits required more, 
not less CRM training. 
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The Success of the Program 

The success of the CRM programs at Continental are shown in Figure VIM (Phase 1 ), 
and in the data from Phases 2 and 3, as displayed in Appendix A, Page A-1 8. The 
results of this probe show an extremely positive attitude toward CRM, compared to data 
we had seen in earlier opinion polls and attitude studies. We are not accustomed to 
encountering attitudes this favorable. It is difficult to pin down just why the Continental 
program in CRM has been so favorably received, especially at a company that had 
recently been through such brutal labor relations. The answer may lie in part with the 
prestige of the instructors responsible for designing and implementing the program. It 
may be also due to the fact that Continental’s programs were always oriented toward 
practical applications in the cockpit, toward procedural compliance, professionalism in 
the cockpit, and down-to-earth presentation of accident reports. Absent from 
Continental’s programs were the “psycho-babble” and “parlor games” that alienated 
pilots in the earlier CRM programs. Recently the generality of CRM principles was 
demonstrated when they were taught to non-cockpit specialists within the company: 
flight technical and maintenance workers, flight attendants, and others (Fotos, 1991). 


C. HUMAN CENTERED AUTOMATION TRAINING (H-CAT) 


The Influence of Delta’s Experience 

In the summer of 1987 Delta Airlines suffered an embarrassing series of incidents and 
close calls. In most cases, lack of communication or ineffective communication 
between the crew members seemed to be the problem. Under the direction of Captain 
Reuben Black, Delta launched an aggressive program to remedy the situation (Byrnes 
and Black, 1993). A CRM program was installed for the entire pilot group, over 9000 
pilots. New hires had their own program. This occurred simultaneously with a growing 
concern over the potential hazards of automation (see Chapter I). One of Delta’s 
incidents involved the inadvertent shutdown of both engines on a B-767 at about 1 000 
feet after takeoff from Los Angeles; but most involved aircraft with traditional cockpits. 
Delta decided to attack the automation issue at the same time as the communication 
problem. A special automation task force was formed, and out of their deliberations 
came a ground school course named Introduction to Avidtion Automation ('colloquially 
7. A. Squared"). 

Students about to transition to a glass aircraft for the first time went through the course. 
It was a model-independent (“generic”), non-technical exploration of the benefits and 
hazards of automation, making use of a variety of graphical material, lectures, TV news 
clips, and accident reports. The aim of the course was to familiarize pilots 
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encountering glass for the first time with the general problems of operating a highly 
automated aircraft, and the company’s philosophical stance. At that time Delta flew the 
following glass aircraft: MD-88, B-757 and 767. Within a short time it would add also 
the A-310 through its acquisition of PAA, and would also obtain MD-1 Is. The 
automation course was considered by Delta to be a great success, and was soon 
imitated by a number of other carriers (Dornheim, 1996b; McKinna, 1996a). 

Corporate Philosophy 

The term “philosophy of automation” became popular, as a result of the work of Wiener 
in automation (1985b), and Degani and Wiener (1994) in procedure design. The notion 
was that the highest level of flight management should spell out a philosophy of 
automation, an over-arching view of the company and how it would expect automation 
to be addressed. From this over-arching view would follow, according to Degani and 
Wiener, policies, then procedures, and the results would be practices (what is actually 
done on the line). They called this approach “The Four P’s”. It became the part of the 
curriculum of the various carriers’ automation course - explaining the company’s 
philosophy of automation. See also p. 11-22. 

Continental’s Response 

Continental’s automation program closely followed the Delta model. It was nicknamed 
the “glass class.” After several name changes it stabilized on H-CAT (Human- 
Centered Automation Training). [We shall use the term glass class in this report to 
mean generically any form of classroom-based automation training, at any company, 
involving generic FMS and glass equipped cockpits. This excludes technical courses 
on particular models.] At Continental, as well as other carriers, the glass classes 
played to mixed reviews, for a variety of reasons. 

1 . Many of the pilots, although encountering glass for the first time, and clearly in a 
position to profit from the model-independent training, were impatient to get 
moving into the “real stuff’ (aircraft-specific systems). Anything that impeded 
this was considered a waste of time. 

2. Classes included (1 ) pilots who were virginal to glass or FMC; (2) former 737- 
300 pilots who were trained in FMC operations, but not glass, and (3) 737-500 
pilots upgrading to the 757 (glass to glass). Most of the pilots with prior FMC, 
and certainly prior glass experience found much of the H-CAT course redundant. 
Most reported that they enjoyed the discussion of the accidents, but felt that the 
rest of the material was a waste of time. 

3. There was a lack of agreement about where in the transition syllabus the class 
should be placed. Some (including the authors of this report) felt that it 
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belonged, as traditionally positioned by most carriers, at the beginning of the 
first day, for a variety of reasons, among them the perceived need to allay fears 
and misconceptions, as discussed in Chapter II. Others in management and 
training felt that the course was misplaced by being offered in the early days of 
transition, before the trainees even knew what the terms meant. They felt that 
the instruction should be at the very end of the transition training, as a capstone 
following ground school and simulator, a kind of “pre-lOE wrap-up.” Clearly a 
case can be made for this approach. 

4. We heard at several carriers that the glass class should follow the systems oral 
examination by the FAA (and designates). Captains, we were told, could think of 
nothing but passing the oral, and were not interested in any instruction not 
provided for that purpose. The statement applies equally to first officers, who 
were also type-rated in CAL’s 757 program in order to give the airline flexibility in 
crewing its ETOPS flights. 

5. Questions about cost of the course and who should teach it were unavoidable. 
Many of the carriers began to reduce the time allotted: at one carrier it went from 
a day to a half-day to two hours, even though the original course was considered 
highly successful. At the same time, questions were raised about the cost of the 
instructors. Cost conscious managers saw no reason why CRM or glass classes 
had to be taught by line qualified pilots, and at some carriers the task was turned 
over to less expensive ground school instructors. This was an offense to many 
of the pilots, and especially to those who had been chosen to give CRM/glass 
class instruction. Perhaps a non-pilot could give instruction in systems. Aircraft 
systems work according to a clear-cut and unambiguous design. So they are not 
difficult to teach. But only a pilot, who had “been there and done that” could 
provide the rich and hard-earned experience needed to discuss actual line 
operations and decisions, crew interactions, and ambiguous situations. 

For example, it might seem ludicrous to have a ground instructor (non-pilot) 
discuss the operational aspects of handling a communication or decision 
problem (e.g. whether to divert due to an ill passenger, and where to go.) Only 
a pilot, many might say, could lead such a discussion. The debate over who can 
teach CRM-type material continues, as airlines struggle to contain what at many 
has become runaway training costs. 

6. It is generally felt that within a few years, the whole idea of an introduction to 
flight-deck automation will be obsolete, since all pilots will have flown glass 
aircraft, even ab initio students right out of training academies (Glines, 1990; Ott, 
1989; Johnston, 1993b). This may be true, but in the short term, perhaps 
another decade or two, there will still be the DC-9 and the 727 crews working 
their way up the seniority ladder, for whom the introduction to automation 
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training will still be necessary and desirable. Telfer (1993) outlines a human 
factors" course for ab initio students. 

One instructor pilot who we interviewed offered this. Glass classes are being 
overtaken by events in the real world. They are going out of style, and need to be 
modernized. And you can no longer tell the pilots the three adages of the age of 
automation: 

1 . Don’t be afraid of the computer. 

2. Don’t let the computer take over. 

3. Don’t fly into a mountain pushing buttons. 

Despite its sometimes rocky history, marked by frequent changes in the course 
materials, pedagogical approach, management techniques, and occasional 
philosophical reverses, H-CAT and CRM still exist at Continental, and in various forms 
at other carriers. As the General Accounting Office report on CRM stated, “regardless 
of the form, the importance of CRM cannot be denied (1997, p. 5).” 

The Politics of CRM Instruction 

CRM committees at most carriers we have visited or worked with seem to live in 
suspension, often distrusted and suspect by others within the company. Perhaps this is 
because they deal with “soft” systems and values, social sciences in an industry that 
admires engineering. CRM committees may be misunderstood and not loved by all. 
These committees do valuable work, often at personal sacrifice, and produce generally 
worthy products. But they are still subject to distrust and political vulnerability. CRM 
revolutions and palace plots are not unusual anywhere in the industry. 


D. THE FUTURE OF CRM AND H-CAT 


AQP 

So far we have said little about AQP in this report. This is because the movement was 
not highly active at Continental during the period in which we were collecting data and 
conducting interviews. Now, with the massive Boeing orders and deliveries, the fleets 
have been working on AQP plans. In the way of an update, we note that all models of 
the 737 at CAL now operate under AQP (-100 and -200 models have disappeared), 
and the 757 will be by July 1999. There is disagreement among the ranks at 
Continental as to the economics of AQP. Some do not believe it will save money at all, 
but may still be worthwhile because of the scrutiny of one’s programs or plans and the 
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self-criticism it requires. Others point to cost-savings in a plan such as the 757. With a 
higher rated FTD, 757 simulator time will be reduced. This certainly translates into 
dollars. Planning for the 777 AQP proposal produced similar results - apparent time 
and cost savings under AQP. 

The savings expected from AQP are elusive. Even some of its adherents agree. They 
argue that an AQP program is inherently expensive, and reducing the amount of time in 
a program does not compensate for this. What AQP does, they state, is force the 
training department to do a better job, but not a cheaper one. We do not take a stand 
on this -- in time there should be figures to clarify this most important point. 

H-CAT 


We envision a future H-CAT that will be lean and cost-conscious, and more “generic” 
(model independent) than its predecessors. Whether this instruction will eventually 
disappear from the training plan, since, as we have suggested, every pilot will soon 
have had his/her one-time inoculation, is largely up to the training departments. To a 
degree this question may depend on the direction that AQP takes in the years ahead. 
Pilots will migrate in and out of seats, as fleets expand in the near future. Training 
departments will be heavily loaded with transition training, upgrades, and new hires. 
The expansion has already begun (Sparaco, 1996; Ott, 1996; Proctor, 1998a). As of 
this writing [summer 1998], Continental had placed orders with Boeing for 92 next 
generation 737s, fifteen B-767-400s, and fourteen 777s. Fifteen 757s were also due 
from an original purchase of 45 (Proctor, 1998a). 

New hires, be they experienced pilots or ab initios , will have already been exposed to 
one or more forms of CRM instruction. The big problem will be to standardize a pilot 
group, who come from a variety of automation and CRM backgrounds and corporate 
cultures. If we had to guess, we would see the programs as no more than a half-day, 
discussion-oriented (rather than lecture oriented), with heavy dependence on accident 
reports, and other materials that can be obtained inexpensively. If taped lectures are 
used, they will be employed to motivate discussion, rather than played without 
interruption in the manner of a classroom lecture. 

There are two questions that we would not place a bet on. (1 ) Whether H-CAT 
programs will be placed at the beginning or end of the transition package (presumably 
not elsewhere); and (2) Whether they will be taught by line pilots or by professional 
ground instructors (who may or may not be former pilots). Airline training departments 
will have to decide when it is time to abandon H-CAT instruction, assuming that it does 
not become required subject matter. We predict that in another 10-15 years, generic 
H-CAT and CRM, as training topics sui generis, will vanish and the worthwhile material 
will be absorbed by the overall transition training syllabus. 
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CRM 


Clearly CRM, in one form or another, has a bright future in the aviation industry, 
although the future form may not be readily recognizable by today’s practitioners. 
Already the predicted expansion beyond the cockpit is taking place, with CRM being 
taught, and we presume applied, in such domains as maintenance, the cabin, and 
dispatch (Fotos, 1991; Helmreich, Wiener, and Kanki, 1993; Helmreich and Merritt, 
1998; Proctor, 1998a). It is not difficult to envision a company, airline or otherwise, 
where all of the employees are trained in CRM techniques, and that communications 
are thereby improved. For this to work, we caution that at least within a company there 
must be consistency of philosophy, approach, and instruction. We would not want to 
see one segment of a company trained on “Type A” CRM and another on “Type B.” A 
CRM culture clash within an organization might be an interesting thing for social 
scientists to observe, but would not be helpful to the organization. The predicted 
expansion of CRM-like instruction into non-aviation fields (Helmreich, Wiener, and 
Kanki, 1993) has already come to pass. We will not pursue that matter here, as it is 
beyond the scope of this project. 

One trend that we expect to continue and expand is for the airframe manufacturer to be 
the provider of the CRM instruction and materials. Airbus Industrie has led this field: 
CRM instruction is part of the “package” when a carrier buys an Airbus. “A pragmatic 
approach to automation is the key element in our training,” according to Captain Pierre 
Baud, senior vice-president for training and flight operations support. This approach 
was the result of the business plan of that company, who saw their products as aircraft 
not only for technologically advanced nations, but for the Third World as well. From the 
beginning, Airbus sought, in their instruction, to understand and manage cultural 
affinities and differences (Sparaco, 1996, p. 133). As part of its training doctrine, 
Airbus will use unusually small training groups, sometimes reaching a student-to- 
instructor ratio as low as 1 : 1 . 

The trend toward manufacturer-provided instruction will no doubt continue. Its success 
may depend on whether crew members can accept instruction that is generic, rather 
than model-bound, generic not only with respect to models, but to company culture. 

We can already imagine the emergence of a joke about the first-day in CRM/H-CAT 
class. The instructor begins his introductory remarks with, “This instruction is designed 
specifically for pilots flying the you-fill-in-the-blank aircraft for you-fitl-in-the-blank 
airlines. The standardization of cockpits of various models, which Airbus has stressed, 
makes CRM, as well as other instruction, easier. There are simply fewer exceptions 
that must be taught. Is keeping instruction “generic” a good thing or bad? Due to the 
costs involved, there may be no choice. The carrier cannot expect to either develop its 
own tailored, model-specific packages, or obtain them from the airframer. 

Finally, we predict the end of the era of “parlor games”, personality inventories, 
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authority grids, and psycho-babble. Perhaps such simplistic approaches were 
appropriate in the early days of CRM, since little else had been developed. But they 
have not served us well. The link between personality and flying duties, 
communication or technical, still is not established except in the extreme. In brief, the 
personality approach was not line-oriented. 

Now, in the age of AQP, the preparation of an AQP proposal to the FAA forces all the 
training issues: CRM, H-CAT, LOFT, use of simulators and flight training devices, and 
others. A 757 training captain put it this way: “We (each fleet) must make our own AQP 
program, and not try to use an adaptation of another fleet’s proposal.” 


E. IN CONCLUSION 


AQP proposals guide the development of the flight training programs. But they must 
not only answer to the FAA, but must as well be appropriate to the carrier’s perceived 
mission, and consistent with the corporate culture. As Degani and Wiener pointed out 
(1994), flight-deck procedures for operating the same piece of equipment vary from 
company to company, reflecting the carrier’s culture. Thus AQP and the training 
doctrine it spawns must be sensitive to and flexible for local conditions, values, and 
goals. The CRM and automation programs must do the same. There is no one correct 
way to design any training program, especially one as culturally sensitive as a CRM 
class. Under AQP, the FAA mandates a CRM program, but stops short of telling the 
carrier what to do (Orlady, 1994). 

It is regrettable that there has been such a lack of creativity in addressing the 
automation philosophies. Every philosophy statement we have seen so far has been a 
rehash of Delta’s. They usually follow this pattern: 

1 . These are the advantages of automation. 

2 This is the philosophy, from which policies will flow. 

3. The company expects every pilot to be able to use the automation at every level. 

4. But the ultimate decision on the level chosen remains with the pilots. 

We hope that in the future some more imaginative work will be done on determining 
philosophies of automation. 
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VIII. DISCUSSION 


A. INTRODUCTION 


In this chapter we shall briefly discuss a number of training concepts, based on what 
we have found from the Continental experience, field observations, and a literature 
review. These discussions are not meant to be comprehensive — more complete 
articles and book chapters are cited. We will include in this discussion the 
administration of training, and the costs of training. Finally we will take a look at 
training technologies of the near future, and how they affect the future of transition 
training. 

At most airlines, training has been highly tradition-based. Usually when a new plane is 
brought into the fleet, the training department operates as it did before, until a dramatic 
change in technology (e g. the jet engine; the swept wing; computer-graphic visual 
displays in the simulator; the FMS and glass cockpit) comes along. This calls for a 
dramatic change in training, a “paradigm shift.” Seldom is there a paradigm shift in the 
training pedagogy or method — the big changes are hardware driven (e.g., visual 
displays in simulators) or regulation driven (e.g. AQP). 


B. COSTS OF TRAINING 


For various reasons, training has become a runaway cost at many carriers. Throughout 
this chapter we shall be concerned with the rising costs of training, and the acquisition 
and use of various training devices. It is difficult to find materials on training costs. We 
recommend Dornheim’s (1992) article on the MD-1 1 training program at Douglas. With 
the rapid modernization of the fleets, airlines are scrambling to buy simulators and 
lease simulator time. They are also acquiring less exotic, though still highly expensive, 
flight training devices, some of which are so sophisticated (and expensive) that they are 
essentially simulators, lacking only visual scenes and motion platforms. 

New Aircraft 

The vast fleets of new aircraft carry with them the high costs of transition training for the 
pilots who will fly them. Some of this will be improved by commonality between models 
of the same company. Airbus Industrie deserves honorable mention for their 
contribution to commonality: their aircraft have very similar cockpit designs and 
configurations, making transition training less burdensome, and allowing a single 
simulator to be superficially changed to allow its use by different fleets. 


VIII-1 



For Boeing, it was more difficult to achieve commonality (except in the 757/767s which 
have essentially the same cockpit -- and same type rating), as they did not have the 
privilege of starting ab initio and designing a fleet of aircraft. The 757/767 was a 
revolutionary design, which had no technological parents in the civilian world. The 
continuation of the Boeing advanced aircraft was the 747-400, a redesign of it's 
traditional 747-1 00/200/300. The -400 had advanced EFIS/FMC systems, and a 
two-pilot flight deck. The 777 flight deck is based on that of the 747-400. The new 
generation 737 aircraft promise a high degree of commonality. The great popularity of 
all models of the 737, as well as the new has sparked a flurry of simulator orders and 
construction of simulator buildings and bays, and a scramble to purchase simulator 
time from various sources. 

Collective Bargaining 

Collective bargaining by the pilots' unions also affects the price of training. 

Increasingly the pilot contracts contain negotiated language about the type and amount 
of training, the time of day at which it can be conducted, and the administration of the 
training. For example, some contracts contain language making performance on LOFT 
sessions “non-jeopardy”, meaning that grades or other performance measures are not 
reported to management, but used only as feedback by the student and the instructor. 

Due to the costs of providing transition training, and the fact that pilots in training are 
essentially non-productive, management attempts to limit the migration from seat to 
seat, sometimes by putting a freeze (i.e. 2 years) on further bids after training for a new 
seat. 

But enforcing this is often more difficult than it appears. With rapid acquisition of new 
technology aircraft, management finds itself breaking its own rules by unlocking the 
freeze process and allowing newly trained pilots to enter new bids. This was carried to 
an extreme in June 1998 when Continental and lACP.agreed on a pilot contract that 
changed the entire pay process to the more conventional “weight differential" system -- 
the bigger the airplane, the greater the pay. A discussion of the longevity-based 
system and its effect on the training facility can be found on page IV-2. As we have 
pointed out In Chapter IV, the change from a longevity-based pay scale to the more 
widely used weight differential scale resulted in a top to bottom rebidding for seats in 
the summer of 1998. It was called a “flush bid". The entire pilot assignment list was 
abandoned and each pilot was free to bid any seat at the airline. We will not discuss 
the effect of the contractual change in detail, as it took place beyond the data collection 
phase of the study. But we here note that the flush bid placed an enormous load on the 
training center, as any dramatic contractual change might. 

For the 757 fleet, whose composition we discussed previously, the change was 
dramatic: the younger-than-usual, technology-oriented captains and first officers stood 
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to be bumped out of the 757 by senior pilots who had been flying lighter aircraft. 
Suddenly the 757 looked attractive as a mid-weight aircraft for mid-seniority pilots, as it 
was at other airlines (Wiener, 1989). The resurrection of the B-767 order made it even 
more so. The only thing that kept the 757 from being more attractive was its reputation 
for poor schedules (from the pilots’ point of view), and the heavy concentration of 757 
lines in Newark, which was not considered a desirable area in which to settle. 

Management Backing -- A Clean Sheet of Paper 

We have written briefly in previous chapters on the importance of management’s 
commitment to quality in aircraft training and its support for new transition programs. 
The phrase we heard time and again from the 757 flight management group was, “We 
had the right people and a clean sheet of paper.” The meaning of this was that the 757 
cadre was free to propose the program they felt was best, unfettered by tradition or 
orthodox ways of doing things. And all important, they had a commitment from 
management that any reasonable request would be supported financially. These were 
important pledges on the part of management: they not only assured the fleet cadre of 
support, but also relieved them of budgetary game playing and financial uncertainties. 
This freed them to design the program that they felt was best suited to the airline and to 
the task at hand, relieved of financial concerns and confident of management backing. 

As it turned out, there was an added dividend for Continental. So successful was the 
757 program that when it became time to prepare for the 777, which arrived in 
September 1 998, that the cadre for that fleet was drawn from the 757. The wheel had 
turned: the 757 fleet managers, who had only three years before been given carte 
blanche to build their team, found the 757 ranks being raided for the 777. 

AQP 


Toward the end of the project we were hearing hints both at Continental and elsewhere 
that carriers that had installed FAA-approved AQP plans found that they were not living 
up to its claims for lower training costs, due primarily to the high “up-front” costs of 
designing a program, obtaining FAA approval, and then implementing and managing it. 
This matter is also discussed in the previous chapter. We are not certain whether this 
view is the result of systematic studies of the costs of AQP, or just opinions. As we 
heard frequently, AQP is “data intensive”. And data intensive is dollar intensive. There 
was general agreement that even without cost reduction AQP was worthwhile, since it 
forced training management to look very carefully at what they were doing, and take full 
advantage of any opportunity to cut costs. Others held that AQP has performed “as 
advertised,” and has been well worth the investment. Only time, and experience 
training under AQP will tell the story. 
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C. FUTURE PILOTS AND FUTURE TRAINING 


Ab Initio and Very Low-Time Pjlots 

It is difficult to estimate, at any given time, the market for new airline pilots. On the 
demand side, it varies with the plans of the airlines to expand their fleets and route 
structure. We say expand because that is the expectation for U.S. carriers for at least 
the next decade (FAA, 1 991 ). The same is probably true for other parts of the world as 
well, with the possible exception of Asia, where major airlines are actually cutting back, 
due to the weakening of business prospects. The economic downturn in Asia may 
indeed be temporary phenomenon and fleet development may resume, though perhaps 
not at the optimistic levels of predictions made in the mid-1990s. Where will the pilots 
for these expanded fleets come from? 

The availability of former military pilots, the traditional source of at least one-half of the 
airline crew members at U.S. carriers, depends in turn on the military budget voted by 
Congress, which impacts two areas: (1) The amount of the budget devoted to training 
new military pilots; and (2) the restrictions on the freedom of military pilots to leave the 
service. Presently military flight training is at a low level; even graduates of the Air 
Force Academy cannot count on a flying slot. About half of the graduating class is 
currently assigned to non-flying posts. 

The regional airlines are another principal source for the majors, but the same question 
can be raised about the regionals: where will their pilots come from? Will the regionals 
be able to keep their young and inexperienced pilots, will the regionals become very 
expensive farm teams for the majors? One should not assume that “regional” implies 
low technology cockpits. Many of the regionals are now flying glass cockpit aircraft fully 
as automated and high tech as those of the majors. 

Another alternative is the ab initio (from the beginning) pilot. (This is discussed briefly 
on p. 11-16. This student has trained, usually at his/her own expense, obtained 
commercial and instrument ratings, and joined the pool of available pilots (see 
Johnston, 1993b; Telfer, 1993; Marino, 1993). The prospect has never been 
particularly attractive in the U.S., perhaps because the pool of available pilots has 
never been that dry (see Glines, 1990). CAL does not plan to recruit ab initio or very 
low-time pilots. 

The usual migration pattern would be to a flight school to serve as an instructor to build 
up flying time, thence to a regional, and hopefully later to a major air carrier. The ab 
initio program may be sponsored by the carriers, as it is in parts of Europe and Asia, 
where there is no other source of pilots, and the carriers must take a hand in training 
them. Unlike the expected migration pattern of new, low-time pilots, these ab initio 
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pilots may go straight from school to a major carrier. Their first duty aircraft could be an 
A-320 or a new generation 737. It is difficult for most of us to believe that a 300-hour 
pilot could handle the duties of a first officer in a highly technical aircraft such as an A- 
320. But the carriers who have tried it say it works well. Fiorino (1 998) describes one 
of those rare cases in which a school went on search of bad weather in which to train: 

Emirates, the international carrier of the United Arab Emirates, will be sending new 
recruits for ab initio commercial airline pilot training to the School of Aviation Sciences at 
Western Michigan University (WMU) in Battle Creek. Until now the carrier has trained 
its student pilots at British Aerospace Flying College at Prestwick, Scotland. Aside from 
the technical facilities and training, other factors in Emirates’ decision to switch its basic 
training included the “more challenging weather”. . . Student pilots will train for 62 weeks at 
WMU’s Pilot Training Center, graduating with a Joint Aviation Authorities commercial 
pilot license and instrument rating. Training will include 1,000 hours of ground school 
and 300 hours of flight and simulator training, including 3 hours in a fully aerobatic 
aircraft. 

Children of the Maaenta Line 

Captain Bruce Tesmer of Continental (personal communication, 1998) speaks of the 
new brand of pilots as “children of the maaenta line. ” 1 His concern is shared by many: 
that these pilots, who go essentially straight from the cockpit of a trainer to the cockpit 
of an advanced FMS aircraft never build the basic airmanship and sense of the 
airplane that comes from working one’s way up through traditional cockpits and finally 
to the FMS/glass cockpit. An example is described by Proctor (1988): “Cadets with no 
flying experience are sent overseas for about a year to receive an initial pilot’s license, 
normally from flight training centers in the United Kingdom. Upon graduation, they 
spend a year as flight engineers in Singapore 747s before moving to the copilot’s seat 
in either Boeing 757s or Airbus A-310s.” Thus the pilot has one year of ab initio 
training, a year at which he/she does not touch the controls, and then finds him/herself 
performing the duties of copilot in a highly sophisticated cockpit. 

This is clearly a legitimate worry. What, the critics ask, would happen if an A-320 
captain were incapacitated. The 300+ hour ab initio pilot would more than have his/her 
hands full. Others dispute this, saying that if such an event occurred, at least the 
inexperienced pilot would have available advanced autopilot modes to help save the 
day. Better that they are flying an A-320 than a DC-9. Only time will tell on this 
argument. 


1 The magenta line refers to an executed course displayed on the HSI. 
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PC-Based Training 


We have noted previously (p. 11-18) the possibility of making use of the growing 
popularity and versatility and declining price of personal computers. If every pilot could 
have his own computer (see also pages IV-6 to IV-7) or at least have one available, 
then these could be networked so that instruction could be managed for the pilot group 
(Nordwall, 1995). We envision a computer network by which pilots could receive 
instruction on their own schedule. They could download and print out portions of the 
lesson if they wished, or merely take instruction from the screen. For a review of the 
potential for PC-based training in aviation, see Koonce and Bramble, 1998. 

PC-based training has high initial costs, but low per-user costs. Tied to the web (see 
below), it would guarantee that all users would have the same version of the program, 
the latest version at that, and would eliminate much of the cost of printing and 
distribution of materials compared to traditional training. It would be easy for training 
administration to correct errors at the central source, and to maintain centralized 
quality control. Dornheim (1996a) provides an example of a PC-based trainer for FMS 
function. FMS training is where it may be needed the most, as this is the area that is 
costly for training using current technologies, and the area where new trainees feel the 
weakest in their IOE and early line experience. A low-cost, PC-based alternative may 
be the answer. Dornheim goes to suggest that one of the biggest advantages is that a 
$14 million simulator is not held up while pilots learn FMS functions and programming. 
A commercial software house provides the system, with a cost (at the time of his 
publication - 1996) of $ 200,000 for a site license. 

A PC-based network would be useful for such activities as learning new systems when 
they appear (e.g. EGPWS), new procedures (e.g. fire fighting and security), and other 
topics normally put off until recurrent training. In fact, recurrent training is a good 
example of instruction delivered at a very high cost. Ground instructors must visit each 
base time and again until all pilots have received the required subject matter, and 
special subject matter for the current year. We believe that the effort previously put 
into lesson preparation could go into preparing software, allowing network-wide use 
and permanent storage of instructional material. 

We would not envision the network as playing a major part in transition to a new 
aircraft, as we recommend in the next chapter that an instructor be available when 
students are involved in CBT activities. The reader may recall that many student pilots 
in the original cadre objected that at Boeing an instructor was not available to answer 
questions or clarify material, or in some cases to confirm that the instruction contained 
an error. In spite of this, there would be some material for transition training that could 
be available on the instructional net, that could reduce the heavy load on the student 
during ground school, a subject of frequent complaints (“drinking from a fire hose”). 
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The development of these materials could impose a heavy burden on the carrier. 
Hopefully consortia of carriers could be formed that would develop and share the 
software. They would have to be extremely careful when the instruction contained 
procedures, rather than just introduction to systems, as even for the same hardware 
and software, each carrier has its own procedures (Degani and Wiener, 1 994), and this 
would be reflected in computer based instruction. To the degree that aircraft could be 
standardized across companies, CBT would be potentially useful and also lower in 
price. But we do not anticipate a widespread movement toward cross-carrier 
standardization. 

Another approach is to have the mainframers play a larger part in development of 
computer-based (and other) material, just as Airbus Industrie has done with CRM 
materials. They have been successful in developing general CRM instruction and then 
tailoring it to the culture of the individual airlines. 

Friendly and Unfriendly Programming of the CBT 


While attending the ground school, we noted several examples where the CBT 
programming was not particularly supportive of the student. Most of these have since 
been corrected, and we will discuss programming and instructional design in the next 
chapter. We include these “unfriendly” cases in this chapter only as examples of what 
can happen in mechanized instruction. 

1 . By and large the CBT is a well-used and well-formatted device. The lessons are 
clear and logical and the way the lesson plan mixes CBT, instructor briefing (2 
hours a day) and fixed base trainer (FBT) is quite effective. Particularly 
beneficial is the manner in which questions/answers that are provided within the 
instruction. Also helpful is the proficiency test at the end of the lesson, and the 
fact that the student must stay with a lesson until mastery on the test. If the test 
is “failed” there is guidance on what areas were missed, and a review can take 
place. We think an effective pedagogy, due to: (1) its ability to aid in the 
transfer of technical information; (2) its ability to focus on the student’s apparent 
weak spots; and (3) its self-motivating property. Everyone likes to do well on 
tests, including self tests, even if there is no immediate reward or punishment for 
performance. 

2. The CBT formatting has some flaws, the biggest being the way it goes backward 
to earlier frames when requested (BACK). The problem is the student has no 
way of knowing how far back it is going to go. Usually the trainee just wants to 
hear the current “frame” again - there is no capability to do this. The student 
should be able to go back to beginning of current lesson, or merely replay the 
current frame. 
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3. The beginning trainee should be given a hard copy of the acronym definitions - it 
is distractive to have to stop instruction and branch to the definitions page. 

4. There is a confusing terminology in the electrical portion of the program. When 
a question is missed, the tape says “the correct answer is in blue.” What it 
should say is “the correct answer is in the blue frame.” Blue is used in the 
electrical lesson to designate AC systems (DC in red), and this results in able 
unnecessary confusion. 

Miscellaneous Notes from Training 

1 . The QRH is not in a user-friendly format. It is difficult to find items. Where 
would you look for a wheel well fire? It is under landing gear, brakes, and 
hydraulics. Maybe all fires should be grouped together. We are not prepared to 
say, so this is not a recommendation. Hot start is in the alpha table of contents 
under “abnormal starts.” No problem if you know that. 

2. Ground instructors we have known love instructional “gimmicks”, which they coin 
and use effectively. One we learned was the instructor’s restaurant analogy for 
MCP/FMA agreement. This is an all-important source of possible serious error, 
entering something other than what was intended into the computer. The 
instructor advised the students: think of MCP as the waitress - you tell her what 
you want. But the FMA is the table . The only way to know if you got what you 
ordered from the waitress is to look at what’s on the table, [simple, neat, to the 
point, and easy to remember], 

Web-Based Instruction 


The locus of instruction must also be considered, with pilots living in far-flung places, 
often commuting around the country to their bases. In days gone by, some instruction 
(such as recurrent) was made available at each base, to avoid bringing thousands of 
pilots to a training center. One example where training was carried out by devices in 
each base was the introduction to TCAS. Companies developed a variety of 
instructional devices, including slide shows and VCR tapes, to provide the training for 
the new device. There may have been computer-based instruction for TCAS as well, 
but we are not aware of any. 

But usually pilots, especially those living at some distance from their bases, pass 
through the base very quickly, with little time for instruction. Clearly the answer is to 
bring the instruction to the pilots’ homes, so they can do it at their convenience. There 
are a variety of ways to do this, but packaging material and sending it out by whatever 
carrier has it perils and limitations. The world wide web (WWW) offers an attractive 
alternative: material could be stored on the web by the training department, and then 
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down-loaded by each pilot into his computer. (For an example, see Hughes, 1998, 
p.65). This would bring new instructional material to the doorstep of the pilot group, 
wherever they may live, and do it in a timely manner. 

Web-based instruction and delivery would allow virtually instantaneous updating of 
materials, and could ensure that all pilots have a current version of instructional 
materials and manuals. It could also allow students to post notice of what they believe 
to be errors in the materials, and obtain rapid corrections from flight training 
management. Likewise, questions could be asked by the students by posting on the 
web. It would be management’s responsibility to monitor the web for questions, 
comments, and presumed errors, and respond appropriately. 

In the next chapter we will discuss conclusions and recommendations for training for 
the high-tech aircraft. Many of our recommendations will involve the training 
techniques discussed in this chapter, and may be somewhat repetitious of this writings 
in this chapter. 
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IX. CONCLUSIONS AND RECOMMENDATIONS 


In this chapter we propose some guidelines for transition training and 
administration, and for appropriate use of automated equipment, with the 
emphasis is on training issues. Our conclusions are based on our work at 
Continental, as well as visits to other airline training centers, and jumpseat 
experience with several carriers. The conclusions and recommendations may 
apply more to one airline than another, depending on their corporate culture, 
present state of training, fleet, type of operations, and many more factors. We 
have tried to be cost-conscious in this chapter, remaining mindful of the price 
tags as well as the benefits of our recommendations. 

A Generalization 

“Training must be considered during the design of all cockpit systems and 
should reflect that design in practice. Particular care should be given to 
documenting automated systems in such a way that pilots will be able to 
understand clearly how they operate and how they can best be exploited, as well 
as howto operate them.” (Billings, 1996, pp. 11-11, 11-12). 

Management Support 

When initiating a new program, particularly a large program such as transition to 
a newly acquired aircraft, management should find the best person for the job, 
and then give him/her a “clean sheet of paper”, meaning put aside the past, and 
launch the program with an open mind and the full support of management. 

(See also p. VIII-3.) Management should never "nickel-dime” the program 
manager. It does not pay in the long run to be over-restrictive with funds, or to 
require excessive justification of expenses. Extra dollars put into a training 
program will pay off in various places. For example, in this study it was clear 
that the 757 program was not only fulfilling its primary mission by turning out 757 
pilots, but as an additional benefit was producing what would soon become the 
777 cadre. This was an unexpected dividend which became apparent when it 
was time to put the 777 program in place, with a very short lead time. 

Curriculum Development and Standardization 

Determine which skills will be taught in ground school, and which will be 
considered “hands on” and left to IOE and line checking. It is very easy for 
ground school instructors to say, “Don’t worry about that - you’ll get it on the line 
(or in IOE).” This attitude can result in serious flaws in the training product, and 
possibly erroneous learning techniques and improper procedures. 
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Likewise, the transitioning pilot tends to hear on the line at many companies, “I 
don’t care what they taught you in ground school and simulator. This is the line 
out here. This is the real world.” 

There must only be one standard, and it must be taught and checked constantly. 
It is a failure of management if a pilot discovers there is a difference between 
what he is taught in ground school and simulator and what he finds on the line. 

All instruction that is given a pilot must be line oriented . What other orientation 
could possibly be entertained? 

We have often advised, standardize the airline, not the pilots . We offer the 
following example. At one large carrier (not CAL) where we did some work on 
cockpit-cabin communication, the flight attendants had a special page in their 
manual that listed the four pieces of information that the cockpit would pass to 
them in an emergency (e.g. signal to brace, time to prepare. The first letters of 
the four formed an acronym. The page was considered so critical that it was 
given unique color and a nickname based on that color. Unfortunately the pilots 
that we interviewed had never heard of the sheet and could not name any of the 
four items the cabin needed. 

It is not up to each instructor to skip over a lesson or avoid questions by 
reassuring the pilot trainee that he will learn this during his IOE. True, there are 
things that can only be learned on the line, such as airport environment, taxi 
procedures, and communication with ground crews. But what the pilot is taught 
and where and when it is taught is determined by the training syllabus, not the 
whim and judgment, however well meaning, of each instructor. Again, 
standardize the airline, not the pilots . 

No single program can optimally serve two sub-populations, one with and one 
without glass experience. We believe there may be some promise in developing 
a semi-generic introduction to glass which is technical instruction material on 
flight deck automation. We envision a pilot taking this course only one time, 
unless there were changes in philosophy at the company. Taking the course 
only once would eliminate redundant training, which is both costly to the 
company and frustrating to the pilot. The answer may be a two-tiered class for 
transition to glass: 

1. Past Glass (pilots who have flown some glass aircraft) 

2. No Glass (pilots who have never flown glass) 

This would allow writing a syllabus that minimizes the re-teaching of materials 
already learned and understood by the first group, and one that starts at “square 
one,” teaching basics of automation and glass cockpits. 
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Documentation and Manuals 


An airline should initially draw as much as possible from the manufacturers’ 
checklist, procedures manual, MMEL, and other documents. But ultimately all 
procedures, documents, and training methods must be tailored to the operations 
and philosophies of the airline, and approved by the FAA principal operations 
inspector (POI). Wherever the material comes from, it now becomes the 
standard documentation of the airline, and the one that is taught and reinforced. 
This is particularly important when an airline goes through a merger or 
acquisition. 

We again recommend that cockpit documentation materials be subjected to a 
thorough human factors study. Specifically we recommend that NASA’s human 
factors experts examine the QRH’s and checklists at various air carriers, and 
make recommendations regarding design and use of cockpit documentation. 

An industry group should examine the products of the “cottage industries,” with 
an eye toward possibly incorporating them into the airline documentation, with 
FAA approval. Flight training management should be curious as to why these 
unofficial and uncertified products are being purchased, when the official 
manuals are furnished free by the training department. One company told us 
their sales of FMS manuals are in the thousands. They must provide something 
of value, in the mind of the purchaser, if airline pilots, with their legendary 
reputation for penuriousness, are spending their own money on these products. 

Training Devices 

[See also pp. VI 1 1-7 and VI 1 1-8 of the previous chapter for a discussion of faults 
and recommended improvements in training devices.] 

While attending ground school we observed that the CBT devices contained 
numerous factual errors and inconsistencies. This propagates false information, 
annoys the pilots, and diminishes the authority of the CBT. Every effort should 
be made to detect and remove these errors before introducing the CBT, by 
testing on a cross-section of the user population, not just ground school 
instructors. Management should be scrupulous about minimizing errors in all 
training software. We also noted several examples where the CBT programming 
was not supportive of the student. Some of these have since been corrected. 

A qualified ground school instructor should be available at all times when 
students are studying via CBT. (Failure to do this was the most frequent 
complaint about CBT training). The instructor can answer questions, and 
resolve differences, when they occur, between manuals and CBT text. 
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Explore the potentials for web-based training, and its ability to centralize the 
timing and distribution of new or revised software. Make it easy for pilots to 
obtain updated software. 

Examine the use of FMC/CDU part-task simulators. These were not available in 
this study. Especially evaluate “free play“ (exploratory learning) on these 
devices, and research whether free play or a more structured approach is the 
most effective utilization of part-task simulators. Attempt to determine the cost 
effectiveness of the FMC/CDU devices, compared to the more expensive FTDs. 

We have found in our interviews and questionnaires a high incidence of FMC 
errors in the first year on the line, These include, but are not are not restricted 
to, failure to arm LNAV after heading selection, confusion over the various 
autopilot-autothrottle modes, confusion over VNAV path and VNAV speed, often 
resulting in failure to make a crossing restriction, and the need to update winds. 
We feel that a part-task simulator with opportunities for “free play” should be 
examined as a potential remedy for these errors. 

CRM Training 

Attempt to shield CRM from “company politics.” CRM training (and trainers) 
seem to be particularly vulnerable to the changing whims of management at the 
carriers we have visited. CRM instructional programs rise and fall very quickly. 
This may be due in part to the fact that these programs are not well understood 
by airline management or the FAA. It may also be due to the fact that the goals 
are vague, and the methods appear to be rooted in psychotherapy, in an 
industry that values engineering over social sciences. The instability of CRM 
instruction may also be attributed to the difficulty in finding “hard” measures of 
success and of value versus cost. 

It is not easy to say how CRM programs can be protected from political whim, 
but we have little doubt that it must be done or the effectiveness of CRM 
instruction will suffer and the costs will increase. [Imagine that at an airline, 
once every few years, there occurred a movement to radically change the 
teaching of hydraulics, and the proponents of the present methods, along with 
their teaching programs, syllabi, and materials were swept away to make room 
for the new!] 

Ab Injtiq and Very Low-Time Trainees 

If ab initio training results in the hiring of low-time pilots for sophisticated 
cockpits, the airline must carefully examine its training program. The training 
appropriate for the more experienced pilot new-hire may be inappropriate for an 
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ab initio graduate. It would be wise for any airline, contemplating hiring ab initio 
pilots to become familiar with the experience of the European and Asian airlines 
which have years of experience with low-time, new-hire pilots. 

Style of Instruction 

There is disagreement in the industry, and among the three authors of this 
report, on where to put CRM and H-CAT in the syllabus. One side says that If 
there is going to be an instructional block on CRM, human factors, or philosophy 
of automation, it should be offered in the first session of the first day in the 
ground school program for three reasons: (1) this is the most effective place for 
introductory material; (2) it prepares the student to "think CRM" from the very 
first session on the FTD and incorporate CRM concepts into his/her behavior; 
and (3) once the student is exposed to aircraft systems, it is too late. The typical 
pilot then does not want to be exposed to anything else. The pressure to pass 
the oral exam starts to build. 

Another equally respectable view prefers to place the systems instruction first, 
let the pilots take their oral exam and put that behind them, and then turn to the 
CRM/H-CAT block, which the student can now learn, freed of “orals stress." 

Develop a culture of helpfulness on the part of instructors toward the 
transitioning students. Discourage any non-constructive behavior on the part of 
the instructors. The “helpfulness” and “friendliness” of the CAL 757 
instructional personnel and the Boeing instructors was commented frequently by 
the pilots in this study, both in questionnaires and in interviews. Alterations in a 
company culture do not come easily and cheaply. 

Administration and Scheduling 


Distribution of pre-ground school information may help reduce apprehension and 
misinformation. Materials could be mailed to each pilot scheduled to attend 
transition training for glass for the first time. Pilots frequently ask for manuals for 
the new plane prior to transition for the plane. This request should be 
accommodated. Perhaps the answer is to provide each pilot on the roster for a 
future transition to glass with a “pre-training package”, of what to expect, a 
syllabus, reading materials, and a schedule of events. Some of our pilot group 
called this a “heads-up package.” 

Pilots should also be encouraged to take a jumpseat ride in the model they will 
be flying, or if not possible, in any glass aircraft, prior to transition training. If this 
is not deemed practical, it may be possible to schedule the trainee to observe a 
simulator session. 
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At some carriers, pilots’ unions have questioned recommendations such as the 
above on the grounds that they constitute work without pay. This is may be a 
valid point which must be worked out by management and labor. We have no 
recommendations on this matter, as labor-management contractual affairs are 
outside the scope of this study. 

Minimize delays between transition training and IOE or line assignment. Also 
attempt to minimize time between transition training and the rating ride. 

Avoid if possible sending newly transitioned pilots back to their old planes. 

Scheduling should provide mixed lines of domestic and overseas flying for all 
fleets that do both. This will keep proficiency for both types of trip at a higher 
level than flying “pure” lines. This is particularly important for those who tend to 
bid only trans-oceanic ETOPS flying, and whose basic flying skill (and possibly 
automation skills) may suffer from flying only very long legs. The use of 
international relief pilots makes this all the more critical. A trip with a few long 
legs and an augmented crew to share the approaches and landings also raises 
the issue of proficiency loss. Mixing domestic and international lines in trips will 
relieve this problem, and probably at little cost. 

Checking and Evaluation 

Explore the potential of using Flight Operations Quality Assurance (FOQA) data 
as a source of feedback for the effectiveness of the training program. The flight 
crews must be in no jeopardy from company or FAA enforcement action. The 
first time FOQA data are used against a pilot, the program will come to a stop. 

Explore the use of electronically based FAQs (Frequently Asked Questions) for 
ongoing design and development of training programs. 

Do not restrict check procedures to full-up automation. Doing so can have 
several negative results, the most critical being that it deprives the crew of 
revealing, and the check airman from observing, the most important aspect of 
automated flying, the crew’s choice of modes and options. Crews should fly a 
checkride as they would any other leg on a trip, and in accordance with company 
policies on use of automation, utilizing the autopilot/autothrottle mode that they 
consider appropriate to the circumstances. If they are not allowed to do this, 
and are required to fly “full-up”, not only are they forced to fly in an artificial and 
perhaps unsafe configuration, but also the check airman is deprived of the most 
valuable data on which to evaluate the crew. 
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The training programs themselves should be subjected to continuous evaluation. 

Encourage feedback from the student pilots and evaluation of all training 
programs, from ground school and simulator through IOE and rating rides, 
including recurrent training. We recommend a formal, structured process for a 
running evaluation of all training programs. The evaluation should probably be 
done by an outside agent, reporting high-up in the training and standardization 
hierarchy. The results of these evaluations must not be mere “number 
crunching” for its own sake, but should affect changes in the training program 
(e.g. curriculum, use of devices, instructional methods). Thus the training 
department creates an instructional feedback loop that should result in 
continuous improvement and quality management in the training program and 
product. 

IOE 

Many pilots reported that the first IOE should not be an Atlantic crossing, as 
there was too much to learn, bordering on overload. They felt that the first IOE 
leg after transition training should be a normal, domestic leg. Introduce 
international flights and especially ETOPS flights only after the “normal” 
domestic IOE has been completed. 

Our analysis of the NASA-ASRS reports (sampled from their entire database, 
names of the carriers unknown) isolated IOE as an instructional phase prone to 
problems, namely competing priorities that could jeopardize the safety of the 
flight. The IOE check airman has two possibly conflicting duties: safely flying a 
revenue flight, and rendering instruction to the newly transitioned trainee. Pilots 
in this study regarded IOE as an important learning experience. However, the 
role of the check airman as an instructor must not be allowed to vie with his 
simultaneous responsibility to fly the aircraft in the safest manner. 

We do not know the answer to this complicated question, and have no 
recommendation except that the problem be studied by both operations experts 
and human factors experts. Obviously the newly transitioned pilot has to start 
somewhere, but perhaps the present IOE structure is not the safest way to go, 
and may not be most conducive to integrating skills and knowledge recently 
learned. Further research should be conducted to identify the specific phases of 
flight and procedures that are most vulnerable to conflicts, and how these 
conflicts might best be handled. Additional research should contribute to the 
design of strategies for the enhancement of line instruction while ensuring the 
safety of flight. 


IX-7 



Early Line Experience 


Some routes are more difficult than others (e.g. more sidesteps, last-minute 
changes in the terminal area, holding, vectors, noise abatement arrivals and 
departures, and more difficult airports). Scheduling should assign easier, less 
demanding routes to inexperienced crew members (somewhat in the same 
manner as less demanding airports for high-minimums captains). We recognize 
that this recommendation places a heavier burden on the scheduling 
department, but we feel that the benefits to training and safety would be worth 
the cost. 

Consider scheduling a one-day follow-up training session at some time during 
each pilots’ first year on the line. The session would be used to answer 
questions, expand on automation techniques, and reinforce little-used or weak 
procedures and skills. A program of this sort would be very expensive, and we 
recommend only that its costs and benefits be considered. 

Cockpit Environment 

Engender the concept of “mutual aid” in line pilots, as contrasted with “separate 
work stations” with only the procedurally required communication. This is 
especially important when one or both pilots are inexperienced (e g. recently 
post-lOE). 

Pilots should not hesitate to reveal to each other on initial contact their 
experience level on the new aircraft. We recommend that this be addressed as 
a form of CRM training. At times great differences in time-in-type will be 
encountered on the flight line. This should be known by both pilots, as one may 
have to compensate for the inexperience of the other. It is probably an 
endorsement of the CRM training that captains we have observed have little 
hesitation in saying to the first officer, in so many words, “I’m new at this (glass 
cockpit) and I’d appreciate all the help you can give me.” Such a statement we 
would regard as good planning, good briefing, and good CRM, and it should 
promote a more relaxed cockpit atmosphere. 

Briefings 


The importance of briefings should be taught and emphasized all along the way 
in transition training. They are the foundation of effective communication and 
the proper performance of duties in the cockpit. Briefings should be 
demonstrated by FTD and FSS instructors, check airmen and IOE instructors, 
and the students should practice briefing in every session. The connection 
between briefings and CRM should be noted. Stress the importance of briefing 
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by the captain (or PF) to the other pilot(s) and to the lead flight attendant. 
Training for briefings should be part of any AQP proposal. 

Flight attendant briefings should also be taught and practiced. Topics should 
include, but not be limited to; (1) management of unruly passengers; (2) 
suspension of service during turbulence; (3) clarification of sterile cockpit 
procedures; (4) anticipated weather; (5) communication between the cockpit and 
the cabin. Captain/flight attendant briefings are a two-way street: the lead F/A 
should brief the captain on any special problems or requirements which he/she 
anticipates for the flight (e.g. heavy passenger load, meal service on a short 
leg.) For a good example of a captain’s F/A briefing, see Chute and Wiener 
(1996, p. 226). 

Briefing of flight attendants should be an integral part of the pilot training 
syllabus and the FOM, and practiced during transition training, including during 
FTD exercises. Annual recurrent training may be a good time to review the 
contents and technique of briefings. 

Planning is closely related to briefing. It is particularly important in two-pilot, 
glass aircraft, and is the foundation of workload management. Planning should 
be emphasized in the H-CAT program, and in FTD and FSS training. The FTD 
is the place to learn and test planning techniques 

Method of Flvina and Flight Safety 

Flight management should formulate a policy on maintaining manual (hand 
flying) skills, and convey this to the pilots. Hand flying of the new aircraft during 
transition training is highly desirable, and some portion of the simulator training 
should be devoted to this. Guidelines for hand flying should be developed, 
specifying where it can be done, under what weather conditions, and what types 
of approaches. As always, the captain’s discretion prevails. We must not forget 
that hand flying can have negative consequences as well (e.g. high workload 
and failure to scan the “big picture.”) 

Allow for the practice of non-automation-based problem-solving skills and 
infrequently used procedures. 

In all flights, observe fundamental rules of safe piloting. Several interviewees 
commented that pilots were not clearing turns in VMC. This precaution goes 
back to the first day any pilot began his flight training. One pilot remarked, “At 
least glance in the direction you are turning.” 
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Consider the FMC/MCP as a control to be “handed off’ to the other pilot, and 
formalize the handoff, perhaps with a callout such as “your FMC,” to provide 
feedback that a transfer has been completed. 

Workload Management 

Management should clarify the policy and procedure on PF/PM duties and 
sharing of work load. It is critically important in the glass cockpit to specify 
clearly “who does what” and to conform to procedures, and to stick to the task 
assignments. [At Continental, the term “pilot monitoring” (PM) has replaced the 
familiar “pilot not flying” (PNF)]. 

We recommend that airlines consider replacing the term “pilot not flying” (PNF) 
with “pilot monitoring,” as CAL has done. It gives the position a more positive 
duty, stressing what the pilot does, rather than what he/she does not do. It 
further enhances the task of monitoring, an increasingly important activity in the 
age of automation. 

Reduce, through systems analysis, the frequency, complexity, and length of 
“company reports", especially at low altitudes or in crowded airspace, whether 
they are done by voice transmission or by data link. Routine calls could be 
made at high altitude, or preferably at cruise. Many calls can be made via 
existing datalinks (e.g. ACARS), but we are not prepared to say that this is a big 
improvement over traditional voice transmissions. Use of ACARS probably does 
solve the problem of frequency saturation that occurs in terminal areas, but 
preparation and transmission of an ACARS message is time consuming. In the 
near future much more sophisticated datalink systems will be available, and 
standard messages can be stored and sent, with a minimum of keystrokes, 
possibly reducing cockpit workload. 

Flight management should examine all company business required of the cockpit 
crew, with an eye toward minimizing, or better yet, eliminating, duties that must 
be performed below 18,000 feet. This recommendation, in one form or another, 
appeared in various places in this study. 

Consider placing some type of terminal, either voice or datalink (e.g. ACARS) in 
the cabin for flight attendants to use for passenger matters. We see no reason, 
other than tradition, why F/As have to come to the cockpit with requests for gate 
information, galley supplies, or wheel chairs. 
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Impact on Future Air Traffic Management 


We have noted throughout this study that pilots are having difficulty with 
navigation, specifically VNAV, in achieving level-offs, meeting crossing 
restrictions, and initiating descents. The FAA should take note of these 
implications for the design of future air traffic management systems such as 
“free flight”. 

Hardware Standardization 


Where possible, standardize over models by the same manufacturer. This will 
cut cost and time required in training pilots, cabin crews, and maintenance 

personnel. It may also enhance the quality of maintenance, and make easier the 
cross-qualification of crews. It may also allow cross-utilization of simulators to 
train for various models. 
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EPILOGUE 


In this study we have examined the development and installation of a training 
program designed to transition pilots from old technology to new, computer- 
based cockpit technology. We have commented on what is good and what is 
not. We feel that the success of the program, its nearly zero failure rate, and its 
acceptance by the pilots is the result of the right people, the meticulous design 
of the program, and the strong support of management. The “clean sheet of 
paper” policy has obviously paid off. 

Pilot opinion is not everything, but measured correctly, it is one valuable 
assessment of the program. Shown below is one of the graphic displays of pilot 
opinion, taken after slightly over a year on the line. The results speak for 
themselves. 



We add only that in prior field studies involving various airlines in the U.S., with 
over 40 opinion probes administered to hundreds of pilots, we have never seen 
a response this extreme on any subject, nor such an endorsement of a 
company’s pilot training program. The authors give credit to the men and 
women throughout Continental Airlines for this success. We hope that others 
will be guided by Continental’s experience, and will not only make use of these 
results, but will make improvements. Aviation safety comes not in dramatic 
breakthroughs, but in slow, cautious, sometimes tedious, step-by-step expansion 
of what we already know. 
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APPENDIX A 


Likert Attitude Scales for Three Phases of the Study 


On each of the following pages in this appendix, the data from the Likert attitude scales 
are displayed in graphic form. There is a plot for each probe, and each phase of the 
study. The page numbers correspond to the probe numbers (e.g. Probe no. 13 “There 
is too much workload below 10,000 feet and in the terminal area” can be found on page 
A-1 3). 

For the first 20 pages (A-1 to A-20), there are three plots per page, in the following 
order: 


PHASE 1 (first day of ground school) 
PHASE 2 (about 3-4 months after training) 
PHASE 3 (about 12-14 months later) 


Thus the reader can compare the graphs over time. For a graphic presentation of the 
phases and sample sizes, see Figure Ill-Ion Page MI-4. 

Probes 21 through 24 were administered only in Phase 2 and 3, as the probe was 
inappropriate for pilots not yet out on the time in the 757. These pages (A-22 through 
A-24 display only those two graphs. 
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APPENDIX B 


AC 

ACARS 

ADF 

ADI 

AFCS 

AFDS 

Al 

AOA 

AQP 

ARINC 

ASRS 

CBT 

CDU 

CFIT 

CRM 

CRT 

CVR 

ECAM 

EEC 

EFIS 

EGPWS 

EICAS 

ELS 

ELT 

ETOPS 

FAR 

FBS 

FFS 

FMA 

FMC 

FMEA 

FMS 

FOQA 

FTD 


Glossary of Abbreviations 


Advisory circular (FAA) 

ARINC communication and reporting system 

automatic direction finder 

attitude director indicator 

automatic flight control system 

automatic flight director system 

artificial intelligence 

angle of attack 

advanced qualification program 
Aeronautical Radio, Inc. 

Aviation Safety Reporting System (NASA) 
computer-based training 
control-display unit 
control flight into terrain (accident) 

cockpit resource management; crew resource management 
cathode ray tube 
cockpit voice recorder 

electronic centralized aircraft monitor (Airbus) 
electronic engine control 
electronic flight instrument systems 
enhanced GPWS 

engine indication and crew alerting system (Boeing) 

electronic library system 

emergency locator transmitter 

extended two-engine operations 

federal aviation regulation 

fixed base simulator 

full flight simulator 

flight mode annunciator 

flight management computer 

failure mode and effects analysis 

flight management system 

flight operations quality assurance 

flight training device 
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GPS 

global positioning system 

GPWS 

ground proximity warning system (see also EGPWS) 

H-CAT 

human-centered automation training (Continental) 

HSI 

horizontal situation indicator 

INS 

inertial navigation system 

IRO 

international relief officer 

IRS 

inertial reference system 

IRU 

inertial reference unit 

LNAV 

lateral navigation 

LOFT 

line oriented flight training 

LOS 

line oriented simulation 

MEL 

minimum equipment list 

MMEL 

master MEL 

MCP 

mode control panel 

MOA 

military operations area 

MSAW 

minimum safe altitude warning 

NATRAC 

North Atlantic tracks 

NDB 

non-directional beacon 

PC 

proficiency check 

PF 

pilot flying 

PIP 

product improvement package (Boeing) 

PM 

pilot monitoring (see also PNF) 

PNF 

pilot not flying 

POI 

principal operations inspector (FAA) 

QRH 

quick reference handbook 

RNAV 

area navigation 

RT 

recurrent training 

TCAS 

traffic alert/collision avoidance system 

TMC 

thrust management computer 

VNAV 

vertical navigation 

WPT 

waypoint 
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APPENDIX C 


Marginal Homogeneity Tests: Q1 vs. Q2 


The eight pages that follow show graphically the results of the marginal homogeneity 
tests, and histograms of the before and after Likert scales. Only the eight probes 
resulting in significant differences on the marginal homogeneity test are included. The 
table on the bottom half of the page displays the homogeneity matrix. 

As explained previously, if there is no change in attitude, the tally would be in the main 
diagonal. Taking for example page C-1 , there were 15 pilots who strongly agreed with 
the probe in Phase 1 , and again in Phase 2. There were 9 pilots who changed from 
“strongly agree” to just “agree". Off-diagonal tallies indicate changes in attitude. The 
further the tally from the main diagonal, the greater the pilot’s change in attitude from 
Phase 1 to Phase 2. For example, on page C-1, in response to the probe (“Flying 
today is more challenging than ever”) four pilots changed their vote from “strongly 
agree” t o “disagree,” a rather extreme change. 

The top graph is similar to the other Likert plots that have been shown, except that two 
sets of data are included in each graph: Phase 1 and Phase 2. These plots give the 
reader a comparison of mean responses from the two phases. 
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APPENDIX D 


Questionnaire Forms - Q1, Q2, and Q3 


The following pages contain the three questionnaire forms (known as Q1 , Q2, and Q3), 
from the three phases of the study (PI , P2, and P3). For brevity the three are not 
shown in their entirety, since there is considerable repetition of attitude probes lists and 
answer forms, as well as repetition in Q2 and Q3 of open-ended questions. 
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NASA/CONTINENTAL QUESTIONNAIRE NO. 1 

Date you started 757 transition: 

MM DD YR 


Name Capt., F/O, S/O 

Home Address 

City 

State ZIP 

Present Base Base after 757 transition 

Home Phone: Area Code and number: ( ) 


Make up an ID code for yourself and enter it below. Use any combination of letters and 
numbers (up to a max of 6). Do not use your Social Security or company pay number, 
birth date, etc. Insert it in the blank below. The characters in the last two positions are 
reserved for our purposes. The full eight characters make up your ID (e.g. 

ELW86815). If you use less than six characters, still include the last two (e g. 
TOM415). 

The red sticker is for you to keep a record of your ID. Please enter your full ID as 
written below onto the red tag and keep it some convenient place. We suggest a log 
book or Jep manual. If you have questions, please call the project director or your 
IACP Safety Committee. Remove and keep the red tag. It's yours! 


Informed consent: 


I have read and understood the material in the attached booklet, including the purpose 
and method of the study, and I consent to serve as a volunteer pilot. 

Signed: 


ID Code: 


1 5 
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I. BIOGRAPHIC DATA AND AIRCRAFT EXPERIENCE 


This is our first effort to collect some information from you about your attitudes toward 
cockpit technology and your experiences. First we need some information about you, 
and about your flight experience. 

1 . Your present age to closest month: years months 

2. Gender (circle one): M F 

3. We would like to know your past experience in Continental turbojet aircraft. 
Please consider your experience only at Continental . Place an "X” in the box for 
each seat on each aircraft that you have ever flown at Continental. Do not put 
flying time. 


AIRCRAFT SEAT 



Captain 

F/O 

S/O 

DC-9 




MD-80 




B737-1 00/200 




B737-300 




B727 




A-300 




B707/720 




DC-10 




B747 





4. Which seat in which aircraft did you occupy immediately before going to 
757 transition. If it is a B-737, please state model. How many months 
had you been in this seat? 

Aircraft Seat Months 

5. Approximate total flying hours at Continental (include S/O) 

hours 
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6. Approximate total flying hours, all aircraft (airline, military, general aviation etc.) 

hours 

7. When did you attend, or when do you plan to attend B-757 transition training? 

Month Year 

8. List in the space below any glass cockpit (EFIS) aircraft that you have flown 
(airline, commuter, military, corporate) 


9. What do you consider the most advanced aircraft (with respect to 
instrumentation, avionics, automation etc.) that you have flown? Include military 
or other employers: 

Answer: 

10. Do you use a personal computer at home? (Y/N) Ans: 

11. If yes, when you are at home, how often do you use it? (check one) 

Daily Several times a week Less than weekly 

12. Approximately how many actual Cat II approaches did you make (as PF or PNF) 
last calendar year? Autolands? 

Cat II Autolands 

13. Approximately how many non-precision approaches (as PF or PNF) did you 
make last calendar year? 

VOR LOC ADF 

14. If the money and quality of trips were all the same, and base assignment were 
not a factor, what would be your first choice of plane to fly in Continental's fleet? 
(Include B-737-500 and B-757 as present fleet.) 

Aircraft: 
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II. ATTITUDE-TOWARD-AUTOMATION SCALE 

This is a 20-item attitude scale. It is called an "intensity scale" because you can 
indicate not only your agreement or disagreement with the statements, but the extent to 
which you agree/disagree. Note that the statements can be positively or negatively 
stated. The scale is straight-forward -- there is no attempt to be "tricky. The next page 
is the answer sheet . 

Answer all questions based on your present experience and opinions with Continental 
aircraft. For the purpose of these questions, consider the word "automation" to mean 
autopilots, autothrottles, flight directors etc. as well as the more advanced flight 
guidance and controls that you are familiar with. 

1 . Flying today is more challenging than ever. 

2. I am concerned that automation will cause me to lose my flying skills. 

3. Automation leads to more efficient, safer operations. 

4. It is important to me to fly the most modern plane in my company's fleet. 

5. As I look at aircraft today, I think they've gone too far with automation. 

6. Automated cockpits require more cross-checking of crew members' actions. 

7. In the aircraft I am presently flying, it is easy for the captain to monitor and 

supervise the first officer. 

8. I am very apprehensive about going through this transition. 

9 Automation frees me of much of the routine, mechanical parts of flying so I can 
concentrate on "managing" the flight. 

10. lam not concerned about making errors, as long as we follow procedures and 
checklists. 

11. I look forward to more automation - the more the better. 

12. I have no trouble staying "ahead of the plane". 

13. There is too much workload below 10,000 feet and in terminal areas. 

14. Automation does not reduce total workload. 

1 5. It is easy to bust an altitude in today's environment. 

16. lam concerned about the reliability of some of the automation equipment. 

17. I am concerned about the lack of time to look outside the cockpit for other aircraft. 

18. Continental's CRM training has been helpful to me. 

19. Some times I feel more like a "button pusher" than a pilot. 

20. I regularly maintain flying proficiency by turning off automation and hand flying. 
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ATTITUDES-TOWARD-AUTOMATION ANSWER FORM 


Referring to the 20 statements, place an "X" in the box that best represents your feeling 
about the statement. Answer quickly -- your first impression is the best. Be sure that 
you respond to all 20 statements. 


■ 

strongly 

agree 

agree 

neither 
agree nor 
disagree 

disagree 

strongly 

disagree 

1 






2 






3 






m 






5 






6 






m 






8 






9 
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10 






11 






12 






13 






14 






15 






16 






17 






18 






19 






20 
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NASA CONTINENTAL HUMAN FACTORS PROJECT 
QUESTIONNAIRE NO. 2 Project ID Code * 



* If you have forgotten your project ID code, please write your name on 
the top of the page - we will remove it and write in your ID code, and send you the 
code. 


This is the second in the series of NASA questionnaires. Please fill it out and return it 
to us in the enclosed envelope. You should receive the next one about a year from 
now. We will take a random sample of our volunteer pilots for face-to-face interviews in 
Houston and may see you then. Again, our thanks for participating in the study. 


Note that the questionnaire has three parts. 

1 . Some information about your present status (on this page). 

2. A 24-item questionnaire on your attitude toward automation. 

3. Four open-ended questions where we ask you to give us some answers in your 
own words. 


Total flying time in B-757: hours. Current base: 

Following your training and IOE, were you assigned to your former plane, or the 757? 
757 Former plane, which was 

If it was former plane, how long was it before you returned to 757? months 

If it was the former plane, did you experience any problems when you returned to the 
757? 

No Yes (please describe) 
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OPEN-ENDED QUESTIONS - Q2 AND Q3 


Please answer the following questions. If you need more space, please write on the 
back of the page. 


1 . What did you think of your training for the 757? Did you have trouble with 
anything? What topics should receive more or less emphasis? Please comment 
on the training aids and devices. 

2. Describe in detail an error which you have made, or have seen someone else 
make, with the automation, that might have led to some undesirable 
consequence. How could it have been avoided? (equipment design, training, 
CRM, procedures?) 

3. What can you say about crew coordination and procedures in the 757? In what 
way are they different from previous planes you have flown? What areas can 
use improvement? 

4. How would you compare the overall workload in the 757 compared to your 
previous plane? Please mention anything that you feel should be changed to 
help you manage workload (procedures, ATC, training, etc?) 


The first four questions were repeated on Questionnaire 3, and the following two were 
added: 


5. Please tell us your strategy for selecting the various HSI modes. Do you always 
use the map mode? For what maneuvers, if at all, do you use the compass rose 
mode? The expanded VOR or ILS mode? 

Can you also give us an estimate of the per cent time for each? 

Map Expanded ILS Expanded VOR 

Rose 

Question is not open-ended. It was added at the request of the 757 fleet manager. 

6. Note: this question for those who have left the 757. 

After you left the 757 and went to another aircraft, what was your reaction? What did 
you miss about the 757 avionics and automation? What did you like better about the 
older technology plane? Why did you bid off of the 757? 

Plane and seat you went to: Aircraft Seat 
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ADDITIONAL ATTITUDE PROBES ON QUESTIONNAIRES 2 AND 3 

Four attitude probes that were not appropriate for Q1 were added to Q2 and Q3. The 
attitude probes on Q2 and Q3 were identical. 

Additional Probes : 

21 . The 757 works great in today’s ATC environment. 

22. Training for the 757 was as adequate as any training that I have had. 

23. Electronic flight instruments (“glass cockpits”) are a big advance for flight safety. 

24. There are still modes and features of the 757 automation that I do not 
understand. 
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NASA/CONTINENTAL HUMAN FACTORS PROJECT 
QUESTIONNAIRE NO. 3 


Project ID Code * 

■ 

■ 

■ 

■ 



1 

5 


* If you have forgotten your ID code, please write your name on the top of this page - 
we will remove it and write in your ID code, and send you the code. 


This is the third and final in the series of NASA questionnaires in this project. Please fill 
it out and return it to us in the enclosed envelope. Again, our thanks for participating in 
the study. 

Note that the questionnaire has three parts. 

I. Some information about your present status (on this page). 

II. A 24-item questionnaire on your attitude toward automation. 

III. Some open-ended questions where we ask you to give us some answersinyour 
own words. There is a special question for those of you who have left the 757 for 
another aircraft. 


I. 

Current aircraft and seat: Aircraft Seat 

Total flying time B-757: hours. Current base: 

Do you “feel comfortable” in the 757 now? Yes No 

If yes, how long after you went on the line did it take? months: 


co\docsVquestVq1 23.app 
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APPENDIX E 


Marginal Homogeneity Tests: Q2 Vs. Q3 


For explanations, see page C-0 
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APPENDIX F 


Continental’s Automation Philosophy Statement (1994) 


The purpose of enhanced autoflight and flight guidance systems in our current and future 
generation of aircraft is to improve precision and reduce workload. 

The pilot must be well versed in utilizing the full capabilities of the automated systems in 
the aircraft. Standard operating procedures for each aircraft have been developed to 
establish conventional methods for utilizing automated functions in any given phase of 
flight. However, it is ultimately up to the judgment of the pilot as to how the automation is 
employed. 

If an automated function improves precision and/or reduces workload in a given situation, 
then its use may be desirable. If an automated function does not complement the 
situation, the use of a more basic mode displays good judgment. 

Continental Airlines expects its pilots to match the level of automation to the flight 
dynamics of the moment. Automated functions are tools. The pilot’s judgment is the 
master of those tools. If automation helps, use it. If it hinders, go back to basics. Match 
the resource to the situation. 
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AUTOMATION COMMITTEE’S RECOMMENDED FORMAT 


The Automation Committee recommended a more graphic format, so that the levels stood 
out in their relative vertical position in the automation-to-manual stack. This design was 
never adopted. 


Continental’s goal for automation is to increase safety and efficiency, and improve 
situational awareness, while reducing pilot workload. Pilots must be proficient in all 
capabilities of their aircraft including the automated systems and must use their judgment 
as to how those systems are employed. Standard Operating Procedures for use of 
automated features have been developed for each of Continental’s fleets and may be 
found in respective flight manuals. 


LEVELS OF AUTOMATION 


I HAND FLOWN 

II HAND FLOWN 

III AUTOPILOT/ 
AUTOTHROTTLE 

IV LNAV / VNAV 


RAW DATA 
FLIGHT GUIDANCE 
FLIGHT GUIDANCE 

FLIGHT GUIDANCE 
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FLIGHT OPERATIONS MANUAL VERSION (rev. 1998) 


This version is included in all flight operations manuals (FOMs): 


The goal of automation in our aircraft centers around safety and efficiency. The purpose 
of enhanced autoflight and flight guidance systems in our current and future generation 
of aircraft is to improve precision, reduce workload, and contribute to situational 
awareness. 

The pilot must be proficient in utilizing all capabilities of the systems in the aircraft from the 
most basic hand flown mode to the full use of the aircraft’s most automated function. 
Standard operating procedures for each aircraft have been developed to established 
conventional methods for utilizing automated functions in any given phase of flight. 
However, it is ultimately up to the judgment of the pilot as to how automation is employed. 


Third and fourth paragraphs same as 1994 version (page F-1). 



APPENDIX G 


Continental Human-Centered Automation Training (H-CAT) 


Simulator Check Airmen CRM Briefing Outline (PC/RT/LOFT) 


Standardized Briefing Guidelines 

The following outline will be incorporated into all simulator PC’s, RTs, and LOFTS. The 
intent is to: 

1 . Inform crews of the recent increase in incidents that have occurred in the past 12 
months and the resulting FAA surveillance program. 

2. Heighten the emphasis on Crew Resource Management throughout the airline. 

3. Elevate CRM from a “nice to know” to a “need to know” status. 

4. Ensure that pilots are routinely exposed to a baseline of standard CRM briefing 
items. 

5. Introduce CRM elements and outline the role they play in all facets of our operation. 

6. Enhance the uniformity of training, i.e. take one step closer to seamless training. 


It is extremely important to include the entire crew in the CRM brief. Instructors working 
with three-man crews will present the CRM brief before splitting up to discuss particular 
PC or RT maneuvers. Active participation by all crew members is critical to the success 
of the program. Utilization of facilitation ana reverse briefing techniques is encouraged 
wherever possible. 


Reference: Enhancing Flightdeck Safety. Continental Airlines Quarterly Standards 
Meeting, April 14, 1994. 
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APPENDIX H 


ASRS Reports Concerning Automation and Training 


1. At approx 90 miles south of SNS on J88 given FL240. Approx 50 miles south of 
SNS given clearance to cross 55 miles south of SJC VOR at or below 1 7000', 25 
miles south of SJC VOR at and maintain 8000'. F/O at this time figure new 
waypoints to make these restrictions and entered them into CDU. Aircraft started to 
descend automatically following instructions given from FMC. Approx SNS VOR, 
OAK Center asked what our altitude was. At that point we were going through 
22000, way above our clearance of at or below 17000' 55 south of SJC. During this 
narrative with ATC the aircraft started to turn left and manually turned back toward 
GILRO intersection. Cause of altitude incursion: descend waypoints entered into 
computer at wrong spot on legs page. Circumstances: Captain checked on aircraft 
two months prior to flight. No flight time given in aircraft. Deemed qualified in 
aircraft by FAA standards. First flight in aircraft since checkout. PC check in 
simulator next day. F/O had approx 1000 hours in aircraft. Was not aware of mistake 
in waypoint insertion in FMC due to lack of experience in aircraft. Conclusion: FAA 
deems pilot qualified on aircraft A model with short course on differences between 
aircraft B. No recent experience qualifications. This aircraft is not the same. The 
aircraft in question should have a distinct type rating for a pilot to be considered 
qualified. Along with this the currency requirements. At present, once a pilot 
completes differences training, he is considered qualified. He might notfly the 
aircraft for a year or more but still be deemed qualified by the FAA. 

Recommendation: separate type rating for this aircraft and the associated training and 
currency requirements. Supplemental information from Accession 71794. We 
missed the crossing restriction by 5000'. I believe we began the descent too late and 
to complicate the situation further, die computer was programmed incorrectly for the 
crossing restriction. I also feel an experienced Captain would have caught the 
discrepancy between the VHF navigation DME and the information given by the 
computer. (Accession #71850) 


2. Experienced failure of one flight management computer (FMC) and had requested 
clearance toward Boston and started in that direction. We then contacted Dispatch to 
find out if they wanted the aircraft at BOS or JFK to replace the failed FMC. We 
knew that if one FMC failed prior to the oceanic gateway, we had to land. But 
Dispatch had another answer, and that was that the flight was legal to continue. They 
mentioned another flight had the same problem a few nights earlier, and our event 
was even agreed to by our maintenance people in Tulsa. They both concurred the 
MEL requirement for 2 FMC's for extended range (ER) operations was for dispatch 
puiposes only. So we proceeded across to Paris. We looked in die operating manual. 
Pilot Operating Handbook, MEL, and could not find any requirement to land if one 
FMC failed prior to the oceanic gateway. Upon arrival at DFW from Paris a few days 
later, we were met by a flight manager who pointed out the references as to the fact 
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that we should have landed. But they were not in the abnormal procedures section, as 
they should have been, but buried in the normal section where we did not look for a 
problem. This procedure has now been moved to the proper section, and the MEL 
will be changed to reflect this. Dispatch and maintenance have been corrected on this 
matter. Contributing to this incident is the fact that the international ground school 
can lead you to believe that the aircraft has 3 long range navigation systems. When in 
fact you only have two. Three inertial reference systems (IRS) and 1 FMC equals 1 
long range navigation system. Those same 3 IRS’s and the other FMC equals the 
other long range navigation system. Failure of either FMC brings you down to one 
long range navigation system. This is not emphasized in the ground schools as we are 
so used to flying with 3 INS, or Omega systems. I am going to recommend that this 
be emphasized in the schools. (Accession #75956) 


3. Our medium-large-transport (non-EFIS) was cleared: ' MUSEC 4 departure, TRM 
transition at or below 3000 ft until 6 DME, maintain 4000 expect FL370 after 10 
minutes.' We set up the cockpit and briefed the departure in accordance with our 
company operations manual for the Santa Ana noise abatement ’ normal cutback ' 
procedure. Distractions were as follows: New captain (first non-checkride line 
departure from Santa Ana), new first-officer (first line trip ever), no APU (requiring 
airstart at gate, and decision from captain to perform noise abatement procedure 
bleeds on (normally it is bled off, with APU for pressurization), unfamiliar aircraft 
(neither of us had much time in the non-EFIS medium-large-transport), unfamiliar 
clearance (specifically the ' below 3 until 6 DME ' part). Our company procedure 
calls for a maximum performance takeoff, flaps 15 degrees (normally reduced thrust, 
flaps 5 ), 28 degree nose up body angle to 1000 ft, then a radical thrust reduction, 
simultaneous flap retraction to 5 degrees, and a shallow climb at 1 / 2+15 to 6 DME, 
on autopilot (vertical speed mode +200 fpm, engaged after thrust cutback). It is a 
challenging procedure even with practice, but for 2 new guys, no APU, new 
cockpit... it starts to add up. Now consider the very heavy GA activity at and around 
Santa Ana on a Sat, and you start to get the picture of the scene in this flight deck. I 
made the takeoff and cutback (to 77.6 % N1 based on an extremely light weight of 
88000 lbs) and attempted to engage the autopilot at 1000 ft as per procedure. Rate of 
climb at this point was well in excess of 6000 fpm (VSI needle pegged). At that 
moment, we were handed off to departure by ATC, and advised to watch for VTR 
traffic 12 o'clock at 3000 ft (typical). I lowered the nose and told my first-officer to ' 
stay outside and keep your eyes open.' The autopilot would not engage, and in the 2 - 
3 secs I spent trying to figure it out (unfamiliar cockpit), the altitude horn went off. I 
thought it was the first alert (2000 for 3000 ft), but in the initial climb we both missed 
that one, and it was the second alert (you blew it!). We were at 3300 ft and our 
ballistic path carried us to 3500 ' at about 4.5 DME. I briskly lowered the nose and 
reduced thrust. ATC asked, 'what was your assigned altitude?' My first-officer 
replied ' 3000 ft..' I added, 'We 're correcting.' ATC answered crisply, Roger.' He 
then cleared us to 13000 ft, and advised us again of Traffic 12 o'clock, 3 miles, 
northbound along the coast at 3000 ft. In my opinion the Santa Ana noise abatement 
procedures are an extreme menace to aviation safety and should be abandoned at 
once. This flight had all the necessary ingredients for disaster: new crew (both 
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captain and first-officer), new aircraft (in the fleet for quite some time but both pilots 
relatively unfamiliar), radical, one-of-a-kind, maximum performance, totally 
nonstandard departure procedures (well practiced in simulator, but done only once 
before by captain on company line check), heavy GA traffic, extremely busy flight 
deck, high deck angle (28 degrees) making see and avoid a complete farce, unfamiliar 
departure clearance. I strongly recommend the following actions: a thorough review 
of all non-FAA imposed noise abatement regulations and procedures. A spotlight on 
Santa Ana in particular (that airport is an accident waiting to happen, ground 
operations ramp, are just simply crazy). Scheduling guidelines that preclude the new 
captain/new first-officer scenario any time the flight is into severe weather or into 
particularly difficult airports. Standardization of airline cockpits, or assignment of 
flight deck crew to only one variant (medium-large-transport A or medium-large- 
transport B or medium-large-transport B EFIS only). Advising ATC to simplify 
departure clearances as much as possible at all times, but particularly when extra 
conditions (weather noise abatement rules, etc.) are imposed. (Accession #99595) 


4. We were cleared to cross 40 nautical miles west of LINDEN VOR to maintain 
FL270. The captain and I began discussing the best method to program the CDU to 
allow the performance management system to descend the aircraft We had a 
difference of opinion on how to best accomplish this task (since we are trained to use 
all possible on-board performance systems). We wanted to use the aircraft's 
capabilities to its fullest. As a result, a late descent was started using conventional 
autopilot capabilities (vertical speed, max indicated mach/airspeed and speed brakes). 
Near the end of descent, the aircraft was descending at 340 Idas and 6000' fpm rate of 
descent. The aircraft crossed the fix approx 250-500' high. Unfortunately, we made 
no call to ATC to advise them of the possibility of not meeting die required 
altitude/fix. This possible altitude excursion resulted because: 1) Captain and F/O 
had differences of opinion on how to program the descent. 2) Both thought their 
method was best: the captain's of programming (fooling) the computer to believe anti- 
ice would be used during descent, which starts the descent earlier; the F/O’s of 
subtracting 5 miles from the navigation fix and programming the computer to cross 5 
miles prior to LINDEN at FL270. 3) A minor personality clash between the captain 
and F/O brought about by differences of opinion on general flying duties, techniques 
of flying and checklist discipline. 4) Time wasted by both Captain and F/O 
(especially F/O) in incorrectly programming CDU and FMS for descent, which 
obviously wasted time at level flight, which should have been used for descent. 
Observation: as a pilot for a large commercial carrier at its largest base, we seldom fly 
with the same cockpit crew member. This normally does not create a problem. I do, 
however, feel that with the "new generation" glass cockpits being on the property 
approx 6 years; this can cause a bit more difficult transition than, say month to month 
cockpit crew change on a 727 or pre-EFIS DC-9. I have flown commercially for 10 
years, and have flown 2-man crew aircraft for 8 of those 10. The toughest transition 
for me is to determine who shares PF and PNF duties. This historically (3 years) has 
been most difficult when the other crew member has transferred from a 3-man 
cockpit to a 2-man "glass cockpit." This is especially pertinent when the crew 
member has been on a 3-man crew aircraft for a # of years. As F/O, when you are the 
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PNF, you accomplish your normal duties. However, often times when one is the PF, 
he also has to do the PNF duties because the other crew member has not been used to 
doing PNF duties to the extent that it is required on 2-man cockpits, whether they be 
conventional or EFIS. This obviously can lead to a myriad of probs. Add weather 
problems or an airport such as Washington National, LGA or Orange county, and 
problems can accelerate with alarming rapidity. (Accession #122778) 


5. Situation: failure to make crossing restriction on Arrival route. The captain was 
flying and I was handling the radios and FMC work. After programming the ATC 
crossing restriction in the FMC we still had about 40 miles before reaching the fix. 

At this time, I told the captain that we were high on the profile and he acknowledged. 

I then began to prepared our landing data and complete the required company 
communications. The captain took over ATC until I could complete my other duties. 
At some point during this process, I noticed that we were still high and I again said 
"we look high to me" and repeated the crossing restriction to the captain. He 
acknowledged and deployed the speed brakes. When I finished the leading data we 
had about 5 miles before reaching the restriction and I then realized there was no way 
we would make the crossing altitude. I told the captain and then informed ATC. One 
factor involved was the captain was new with only 35 hours on the narrow 
body/widebody. I have over a year on the widebody, but this was my first [narrow 
body] trip. While the narrow body/widebody are considered to be one "type" by the 
FAA and my company, they have many differences. One of these being performance, 
so when I thought we were high, we were really very high, and the action taken by 
the captain to correct the descent path, which I thought would be adequate, wasn't. 
Along with this, the training I received for the [narrow body] was inadequate. 

Ground school consisted of 6 hours of [narrow body] differences that was nothing 
more than "filling the squares" for the FAA. We didn't even have all of the [narrow 
body] revisions for our widebody manuals at the time, and there wasn't any required 
flight or simulator training for the first officers. Other factors include my distraction 
away from the descent profile in the form of company communication, preparation of 
the landing data and informing the cabin crew of our position so they could prepare 
the cabin for landing. At the time I thought I was still in the "loop" enough to be 
aware of what was going on, but obviously I wasn't. Also maybe if I had told the 
captain that it didn't look we would make the crossing restriction instead of saying we 
looked high he might have made a greater effort to get down. (Note: after years of 
operating widebody aircraft, my company has just started adding the [narrow body] to 
the fleet). (Accession #124072) 

6. Departed SFO on runway OIL and tracked SFO 350 degree right per SID. Checked 
in with departure control per Tower. Departure Control cleared us to 210 degree 
heading and 4000 feet. F/O (PNF) misunderstood heading and a few seconds 
attention over the correct heading was taken by both pilots, the captain (PF) did not 
hear the new altitude and did not notice F/O reset altitude in mode control panel from 
previous cleared altitude of FL230. The aircraft was climbing at approx 4000 fpm 
and by the time altitude deviation was noticed by F/O an altitude of 4400 feet was 
momentarily attained before return to 4000'. 2 factors, I believe, underlie this 
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deviation. The first is that this was my first trip in the aircraft since "shotgun". I am 
also new to the "glass cockpit" and feel that a certain amount of my attention is 
occupied in interpreting the glass cockpit instrument presentation, in this respect, I 
feel that having only had 4 simulator rides in training with the glass cockpit is 
insufficient, although it is FAA approved. This does not develop enough proficiency 
to apply in the real world with the multiplicity of contingencies that can and do occur. 
The second factor is that just prior to the altitude and heading clearance, I had called 
for an infrequently used but SOP autoflight function. This caused the F/O to stop and 
consider his response and thereby interrupt his "flow" during this busy time of aircraft 
configuration cleanup, frequency changes and reception of communications. My last 
remaining thought is that I should be more decisive during my preflight and brief the 
F/O more thoroughly on what to expect from me in the way of post takeoff 
procedures. (Accession #125079) 

7. While climbing after takeoff to 13,000' we "overshot" the assigned altitude by 500' 
(13,500) and immediately leveled back to 13,000. Related factors: both pilots type 
rated [narrow body] widebody. Both pilots initially trained on and experienced on 
widebody. [Narrow body] and widebody flown interchangeably by same crews. In 
that [narrow body] and widebody are common type rating, once having the initial 
check out in one no further aircraft checkout required for the other. It’s possible for a 
legal [narrow body] crew to have never flown the aircraft and be assigned a revenue 
flight! This was my 4th leg in the [narrow body]. This was the captain's 1 st! While 
it is extremely common [narrow body] to the widebody there are subtle differences 
that are distracting if it's your first encounter with the [narrow body]. In this case I 
missed my 1000' before level off call because I was distracted by either being 
assigned or mis-selecting the appropriate radio frequency. Keeping in mind that the 
light [narrow body] is climbing at 500 fpm. I'm not used to this. The captain surely 
wasn't. And for his first flight in the airplane to be climbing at 400 fpm in clouds and 
snow with copilot that is also new to the aircraft, is only stacking the deck against 
yourself. In essence I was "given" the job of "checking out the new guy" but I don't 
have enough experience in the [narrow body] to do this and watch everything else. I 
would most strongly urge that we return to the policy of sending a check airman with 
each Captain for a few legs. Let the new guy work out the kinks with someone on 
board that is trained, comfortable and familiar with watching the whole operation and 
the other pilot should the need arise. (Accession #129814) 

8. At cruise altitude captain went to restroom. Clearance was given to cross a fix at 
19,000' and to change to another frequency. The frequency and 18,000 were read 
back. At this time the captain returned to the cockpit. The changeover freq. was 
dialed in but the new controller was not contacted. With the help of the captain the 
crossing restriction was loaded in the FMC and it was determined that speed brakes 
were needed to make the restriction. On level off at 1 8,000 we were told to return to 
19,000 and contact the controller as originally instructed. We were in VFR 
conditions and were informed by the controller that no other aircraft were in the area 
at 1 8,000. The F/O just completed IOE on the [wide body] and was an engineer for 4 
1/2 years prior to this. I feel that an instructional tape describing dos and don'ts of 2- 
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man crew everyday operation might have some assistance for a crew member 
transition from a 3-man crew. (Accession #132083) 

9. Check airman in right seat of glass cockpit aircraft, instructing on Captain’s IOE, was 
trying to maximize instruction time in all phases of flight. During later stages of 
descent, discussing descent in VNAV and other modes, ATC issued a clearance to 
cross SEAGO waypoint at 11000 ft and 250 knots. Shortly thereafter, we received an 
off course vector and, after another brief period, a vector to intercept the Arrivals 
route. Check airman, as PF, did not arm LNAV for the intercept as it was close to 90 
degrees, which would result in a rapid course capture and a more radical turn than one 
might wish. The plan was to turn to intercept using heading mode and capture LNAV 
when closer to track. Further discussion about the aircraft distracted both pilots and 
they flew through the Arrivals route. The vectors had also interrupted the descent 
profile. The ATC controller called to ask whether we intended to intercept and with 
this 'wake up' call we did reverse and intercept, abeam SEAGO descending through 
13000 ft. Instructing in the niceties of the aircraft had taken our attention from the 
business of flying the clearance. After landing on the west side of DFW, we were 
cleared to cross the inbound parallel runway, cross a bridge and contact ground 
control on freq. xxxx. We crossed the runway, crossed the bridge, but check airman 
got busy again and didn't call ground control until about half way to a distant gate. 
Fortunately, this was a low traffic period and there were no apparent conflicts. The 
moral is obvious: pay attention to die business at hand. The luxury of the third crew 
member is no longer available in recent aircraft (excepting long haul) and more 
diligence is required of both pilots. It is much too easy, and not uncommon, for 
instructors to become involved in extolling the virtues of 'gee whiz' airplanes versus 
*no whiz' ones and not devote the necessary attention to precise operation. (Accession 
#184917) 

10. Aircraft was discovered off course to the north. A turn south was made. Shortly 
thereafter, I checked the 'cross-track' on #2 INS and found it to be 20 miles left. I 
would estimate total off course error was 25-30 miles. Intending course at the time 
was westbound from 57 degrees north 30 degrees west to 55 degrees north 40 degrees 
west. Autopilot was discovered to have defaulted from 'nav' mode to 'heading' mode. 
Heading bug was set approx 30 degrees right of intended course. Aircraft was plotted 
on course at a position of approx 56 degrees 40 min north 32 degrees 00 west about 
10 mins prior to incident. Autopilot will default from 'nav' to 'heading' during a 
course transfer on EFIS course/heading panel, but this function wasn't accomplished, 
so I have no idea how autopilot got to heading mode. Normal procedure calls for one 
INS to remain in 'course' page for a readout of track error (distance from track 
centerline) I was conducting IOE training at the time and going over various 
functions of both INS units; catalogs, data pages, etc., so neither INS was in course 
page. On analog type aircraft HSI needle would be full-scale deflection before being 
10 miles off course. On EFIS presentation in 'map' mode and 600 miles scale 20 
miles off course is hardly noticeable. My flight director was showing a command to 
turn left, but this is not unusual when the opposite side INS is controlling the 
autopilot. On analog aircraft if the autopilot defaults from 'aux nav' (INS controlling 
autopilot) it goes to 'turn nob' -- which maintains current heading. On EFIS aircraft a 
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default goes to 'heading' which, depending on bug setting, can cause a rapid departure 
from intended course, as happened to us. I believe increasing crew awareness to the 
fact that an 'off-course' situation will not be displayed in the same dramatic fashion on 
EFIS aircraft as on analog type displays is important. I also believe crews 
transitioning to EFIS equip should be aware of the importance of scanning ADI nav 
mode displays which show what is controlling the autopilot My own inexperience 
on EFIS aircraft (approx 100 hours) contributed to this incident. (Accession 
#223697) 

1 1. 1 was on the first leg of my IOE on a widebody. (My first flight of the aircraft.) The 
flight was LAX/ATL. As we began the Rome Arrival, ATC asked us to cross 50 
northwest of Rome at FL290, descend to FL240. I tried unsuccessfully to enter the 
restriction in the FMS. After 3 attempts, the captain tried unsuccessfully and tried to 
explain why it wouldn't take it. Meanwhile, no descent was started. Captain said 'just 
descend manually, I'm going for the ATIS.' However I descended, it wasn't fast 
enough, especially with a 70 kt tailwind. As we neared the 50 NW point from Rome, 
the controller became more concerned and asked if we understood die clearance, and 
what our DME was. I said we understood, but didn't reply regarding to DME because 
I wasn’t sure what he meant, from where. As I was calculating distances, he came 
back and asked if we could be out of FL290 in 30 seconds. We were just under FL 
310 at the time and I said yes. We made it, continuing down to FL 240. We had the 
conflict aircraft in sight for about 43 seconds to a minute, as we went by 1500 ft - 
2000 ft above us at about 11-1 o'clock. The captain was back in the loop at about 
the '30 second' request, and Captain also set FL240 in the MCP, as that was missed 
when the clearance was first received. It is very clear to me what the problem was 
here. A big part of it was my first leg flying the airplane, but also, we are flying an 
airplane, not a computer. My focus on the FMS got in the way of my doing a very 
simple aircraft descent profile. I will be focusing on flying first, programming 
second. Supplemental info from Accession 259900: When I diverted my attention to 
getting the ATIS we had approx 30 miles to go to the crossing fix and to lose 6000 ft 
this was not a problem. However, a very slow descent occurred due to the F/O’s 
unfamiliarity with the descent features in the FMS. One problem here is the training 
in ground school and the simulator doesn't always give a pilot a good knowledge of 
descent features of the FMS due to the fact that most of the simulator work is pattern 
altitudes. To the extent possible, this check airman from now on forward will not 
assume that the new pilot will yet have a good operating knowledge of the equipment 
and when those times that require diversion of attention to other duties to keep a 
closer eye on the aircraft. (Accession #259889) 

12. This was my first trip on this aircraft without training people aboard. This is still a 
brand new aircraft and none of us pilots have had much exposure or experience flying 
in it. We were on the CIVET profile descent to runway 25L at LAX. Our crossing 
restriction was 14000' to CIVET. We misinterpreted our instruments and began 
descent to 10000', believing we were inside CIVET. At about 13000' the LAX 
approach controller told us that we started down early and needed to maintain 14000' 
until CIVET. We immediately climbed to 14400', the assigned altitude to CIVET. 
After rechecking our instrument we realized that our DME reading was based on 
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FUELER intersection instead of the LAX localizer DME. I feel this was an easy 
mistake to make considering our very limited exposure to this aircraft. I find the 
glass cockpit a very difficult system to master and a frightfully easy way to make 
critical mistakes— at least when the pilot is new to it. The problem occurred when 
both of us mistook the DME for FUELER intersection displayed on the nav display 
for the LAX DME, a smaller font image on the primary flight display. The fix for 
this problem, I believe, is more training for the crews. Checkouts have become 
extremely costly forcing airlines to make them in the shortest time possible, which is 
understandable. However, I think more training would help pilots with this extremely 
complex new flight system. Callback conversation with reporter revealed the 
following: reporter cites that this was first trip w/o a check airman on board. Also 
states that this flight crew was very low on combined experience as the captain had 
only 30 hours of experience. That is counting the 25 hours obtained on IOE time. 
Reporter also states that the 1 5 hours he had as operating experience was 3 takeoffs 
and landing's and the rest of the time was logged from the jump seat. Reporter feels 
that this is too little exposure to the real world of operating a $125,000,000 aircraft 
and that he was overworked in the Arrival and got confused as the captain started the 
descent prematurely. He was of no assistance in preventing the deviation. This event 
occurred in spite of 3 years and 9 years operating time on standard 747's for F/O and 
Captain. It could be suggested that if at all possible, 2 low time pilots should not 
mixed together as a flight crew. The coordinated crew concept suffers from the 
composite low experience level and exposes the aircraft, crew, and passengers to an 
unnecessarily high risk of incident, deviation, or accident. The economics as 
practiced in this low training hours approach cannot be justified considering the 
possible results from a mix of unfortunate circumstances being thrown to a set of low 
in type pilots in an ever changing and ever increasingly complex environment. 
Providing the best in hands on experience and training should be the goal and it is 
considered, from a historical viewpoint, that F/O's should obtain their operating 
experience in the seat that they would normally function and therefore be of more 
assistance to the PIC. Jump seat riding should not be considered for operating 
experience in this complex aircraft. Callback conversation #2 with reporter revealed 
the following: the primary flight display (PFD) was on "ILAX" showing ILS/DME 
distance from 25L at LAX. Nav display (ND) showed mileage to waypoint in stored 
route. Reporter could not explain why mistake was made when all the waypoints 
were in the stored route of the FMC. The FMC system auto selects the required radio 
for nav display with, in this case, the 25L "ILAX" ILS/DME being selected. The 
ILS/DME, according to reporter would not be auto-selected automatically until about 
30 DME out unless "forced" through selection and activation of certain push buttons 
near the screen. The "time" attached to CIVET waypoint was not considered in the 
election for descent. The "bottom line" in the assessment of this event is training and 
the amount of technical expertise that is introduced to the student in that training 
atmosphere. There is a level of certainty in the future of the "glass cockpit" and its 
portrayal of valuable, usable data. This however comes about through repeated use 
and experience. Initial training that disallows hands on use in the "formative hours" 
can only be previewing another altitude deviation or misinterpretation that may have 
more serious considerations. The potential for error in a low time flight crew must be 
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re-emphasized as an evaluation is made of further comments from reporter. On this 
aircraft there is no ACARS system thus requiring the PNF, in addition to his other 
duties to contact the company with landing ETA and gate info. Add to this, on a 
"CIVET STAR" the fact that LAX airport constantly uses the task inducing procedure 
of runway switching to facilitate aircraft departures. Consider the additional 
workload to re-program the FMC by getting into the pages of the CDU and selecting 
the newly assigned runway/ILS for approach. Proper crew coordination would then 
dictate another task induced approach plate review. (Accession #307372) 
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